Even under a veil of silver cloud, raindrops and choppy surf spraying up in our faces, there isn't a more exciting holiday transfer than a speedboat. As we bump and squeal our way from Split airport to Brac, my 9-year-old hangs on to the squashy seats as she screams: "We've had 90 big bumps so far!" The driver asks if we want to go faster, as he puts his foot down for the last 20 minutes of the half-hour journey.

We arrive in the tiny town of Milna, which is known for being the most beautiful and safest harbour in Brac. It's springtime, so there aren't the bustling crowds that summer brings, although it is always less touristy here than the neighbouring island of Hvar.

Tall sandstone buildings overlook the water, many several stories high. Some are bruised, battered and boarded up, while others have been renovated into holiday flats or luxury villas.

It doesn't feel like there's a lot of money here, I note, while watching a young man trying to stop his boat from capsizing by scooping up buckets of water and pouring them back into the sea. Paint peels off shanty shutters and rusty wrought-iron balconies call out for a clean against the shiny new boats in the marina, but that only makes me love it more. The ragged edges of the harbour are perfectly picturesque, and the bobbing boats create a feeling of calm.

We shelter from the April rain inside our big, shiny villa, which sits just next to the waterfront.

The sleek furnishings all scream luxe, in a super-understated way, and the five bedrooms blend earthy brown woods with fresh white linen on big, comfy beds, with grey tones throughout. All the rooms bar the bunk have an en-suite with a rain shower bigger than most fancy hotel rooms, and there's a bath for the kids (and me), too.

On the lower level, there's a small gym for anyone wanting to work off that holiday ice cream and gin habit. (The bay makes an excellent running track for those who prefer an outdoors workout.) There's a sauna, washing machine and tumble dryer, and the upstairs open-plan set up is really sociable, with two turquoise velvet chairs just begging for an afternoon snooze.

The pool is small but lovely, with sunloungers, an outdoor kitchen and more speakers. There are obvious disadvantages to coming out of peak season. But even a freezing cold pool doesn't put little ones off. We did pack wet suits just in case, and they are definitely needed, but it's almost more fun splashing in water that's too cold for grown-ups to contemplate. Even daddy won't go any further than a shin dip.

All this obviously comes at a price, but that's another plus of visiting outside of summer - you can save yourself two thousand pounds on a week's holiday here.

Walking away from the harbour and up into the hills, giant, spiky cacti and aloe vera plants line the rough tarmac roads. Thirsty olive trees reach into a dirt track as we meander further upwards towards a pretty olive grove watched over by a herd of sheep, their bells ringing faintly in the distance.

Stones piled on top of each other form crumbling, makeshift walls and new life can be seen in furtive fig trees, the bright green leaves all fresh and new, with tiny fruits peeking out from behind.

The scent of pine trees fills the mountain air and moth nests, made by the pine processionary species, hang in the branches, like papier-mache spiderwebs.

There are lots of cycle trails here - all a little challenging for Poppy and Rosie, aged 6 and 9, but great for teens and adults. Instead, the girls collect dense, dark pine cones on our walk, to mark a trail for our return.

An old man insists on trying to chat to us, despite his lack of English and our non-existent Croatian. He tries to crack jokes we pretend to understand, and makes cat noises and pouncy gestures at the girls. Their frowned faces soon unravel when he gifts them a fresh orange though, which they look forward to juicing in their mocktails later.

It's good to take kids to a place where life is simple. It might be all plush surroundings, Netflix and Sonos sound systems in the villa, but out on our walk, we're discovering nature - the buzz of the bees filling our ears, while butterflies chase each other in the breeze. And Rosie's as happy scrawling her name and making emoji faces in the dirt tracks, as she would be playing computer games. It's a slow pace of life, much needed as we escape the pressure and busyness of the work-school-clubs-homework-socialising hamster wheel, we seem unable to hop off at home.

Our reward for all that walking rests in the middle of Milna. The fast food shops might dish up burgers and chips come sunset, but by day, there's an array of ice creams and sorbets for sale, all whipped up into creamy peaks. For 36 kuna we buy a cone each (the chocolate ice cream is so rich, you might want to sink your whole face into it) - which works out at about €1 a pudding, and puts a smile on all our faces.

When it rains, it can feel a bit like a ghost town, but once the sun's out, locals fling open their shutters and hang washing from their balconies. Kids ride bikes around the harbour and the bars you thought had closed down are suddenly open, serving pints of Karlovacko beer on their outside tables and chairs.

Food and drink is incredibly cheap here. Two filo pastry tarts - one feta, one meat - from Perkana bakery costs 27 kuna (€3), and feeds all four of us for lunch with leftovers. Dinner at Pizzeria Slika sets us back 396 kuna (around €40) and includes three enormous, freshly cooked pizzas, all dripping with hot, stretchy cheese, plus a piping-hot garlic bread pizza, the biggest pitcher of beer on the menu, an Aperol Sprirz, Coke and bottle of water.

At night, street lights ripple in the water like multiple golden sunsets, while tatty old fishermen set themselves up at the water's edge, hoping for tonight's dinner to come swimming their way.

Venture along the coastal road and the landscape turns lush and green. The road gives way to a rusty red track, anchored by grey rocks, bushy green pines and a sparkling sea. Or hire a car and seek out Bol, the Golden Cape Beach, Pucisca and Lovrecina. There's so much, yet so little, to see and do here, it really doesn't matter when you come.

Out-of-season rainy mornings make for lazy starts and catching up on much-needed sleep. The urge to stay inside means your exercise routine doesn't go awry and finally, you get to focus on some family self-care. Snuggling up with our small people and watching Our Planet becomes a daily ritual, and we even book a therapist to come to the villa to give every single one of us a massage (500 kuna (€60) for four half-hour treatments).

Reading a brilliant book in a hot bubble bath is (almost) as relaxing as lounging on a sunbed by the pool. And when you're in a huge villa, the kids are having so much fun playing in new spaces, that not only are there zero interruptions during your hot soak, there's also time to slather yourself in body lotion afterwards.

And when the sun comes out, you really appreciate it. There's no rush to fit everything in; you take each day as it comes, see what the clouds are doing and make your decisions from there. It makes for a really relaxing break, and one that many families (mine especially) need in order to de-stress from the bustling busyness of modern life.