Forem: brk

Oh, the water, the water!

brk — Mon, 03 Mar 2025 17:26:59 +0000

The data is out, the challenge has started..

Hackaton rule number 1: Allow yourself a headstart by carefully lurking for any signs. Will you crack the enigma of the flyer's cryptic clues?

Source: Toulouse DataViz

Well now, the good folks over at Toulouse Dataviz, they've really gone and done it this time. They've put together a whole mess of datasets to commemorate that doozy of a flood back in (eighteen)'75 the one that just about tore the heart right out of this town. 200 souls lost, 25,000 left without a roof over their heads. To them, it was the end of times, I tell you.

But the previous generations had the wisdom to set up a system for monitoring the waterways and rainfall. For more than a century, thousands of little hands have recorded this data, day after day, come what may. Rain, wind or snow, they spared no effort to keep a record of nature's caprices.

This data is now available here in various formats, so that everyone can dive into this story of water and resilience - it's all there to remind us that this river has always been a force that is both beneficent and destructive.

Hydrometric stations:
    * Historical records of 132 hydrometric stations, including details like station name, river, opening/closing dates, coordinates, and more.
    Time series data on water levels during 9 major flood events from 1857 to 2022.
    * Time series data on water discharge during 5 major flood events from 1905 to 2022.
    * A long-term, high-resolution (hourly, 30-minute, 15-minute, 5-minute) water level series for the Toulouse Pont-Neuf station dating back 160 years.

Precipitation:
    * Rainfall data from meteorological stations, collected around the time of each major flood event.

Terrain Elevation:
    * High-resolution elevation data for a 100-meter buffer around the Garonne River basin.
    * Detailed elevation data focused specifically on the Saint-Cyprien neighborhood of Toulouse.

Make no mistake, friend, that river, she's a wild one. A force of nature that can give life one moment and take it away the next. But with all this information at our fingertips, maybe, just maybe, we can start to tame her a little. Learn her ways, predict her moods, and keep the heartbreak at bay when she decides to flex her muscles again.

It's a tall order, I know. But hey, if those folks back in the day could do it, with nothing but pens and papers, then by golly, we've got a fighting chance. So let's roll up our sleeves and get to work, shall we? The Garonne's got a story to tell, and it's high time we listened.

Hackaviz 2025 github repo.

Datasets are available here.

Rules and presentation of the challenge are available here.

In short:

You can use any tools
No obligation to use all the available data.
Aside from external map tile layers, candidates are not authorized to use any other data than the provided one but it's authorized to perform any type of calculation on the provided datasets.
The submission format for static visuals is PDF. In the case of an interactive data visualization, the submission will contain the URL and the deliverable must not exceed the equivalent of 2 A4 pages or the equivalent of 3 screenshots.

(Cover picture: Pont d'Empalot, Inondations de 1875 by Eugène Trutat)

The Curious Case of the Visualization State Machine

brk — Sun, 02 Mar 2025 23:39:56 +0000

It was a curious thing indeed, this state machine. I stumbled upon the concept quite by accident, you see, while tinkering with some coding challenge. At first, it seemed like a jumble of gears and levers, a confusing mess of switch-statements and method calls. But the more I studied it, the more I began to see the beauty of its design.

Now, a state machine - what is it, you ask? See it like a well-oiled machine. It breaks down a great, swirling mass of computation into neat, tidy little states, each with its own set of rules and transitions.

You see, this machine is no ordinary automaton. It has states, like when you have different rooms in a grand old house, each with its own unique character and purpose. It also has transitions, just like the pathways of that house that are allowing you to move from one room to the next.

The state machine in action

Some more findings:

One state is defined as the initial state. When a machine starts to execute, it automatically enters this state.

Each state can define actions that occur when a machine enters or exits that state. Actions will typically have side effects.

Each state can define events that trigger a transition.

A transition defines how a machine would react to the event, by exiting one state and entering another state.

A transition can define actions that occur when the transition happens.

Actions will typically have side effects.

And what some of us might call a finite-state machine (FSM), are the foundation of all computation. They are the building blocks of everything from digital circuits to software programs. By breaking down a problem into a set of defined states and transitions, we can create systems that are more predictable, maintainable, and therefore scalable. It's like taking a great, tangled ball of yarn and carefully, methodically, winding it into a neat, tidy skein.

Now that you've had a look at the big picture, watch a live demo of a (basic) state machine tailored to our data visualization use case and then, we'll dig further.

A never-ending mechanic

Take, the updateVisualization() method. This is the heart of the state machine, the conductor that orchestrates the entire performance. It is a simple enough method, really, just a switch statement that calls the appropriate state-specific method based on the current step:

/**
 * This is the State machine.
 *
 * Updates the visualization based on the current step.
 * @param {number} step - The current step of the visualization.
 */
updateVisualization(step) {
  const visualizationSteps = {
    0: () => this.showInitialState(),
    1: () => this.showIntroState(),
    2: () => this.showZoneState(),
    // ... and so on
  };

  visualizationSteps[step]?.();
}

The line of code below simply means that the first state of the visualization will be handled by showInitialState().

0: () => this.showInitialState()

Within each of those state-specific methods, the real magic happens. Take the showZoneState() method, for instance:


showZoneState() {
  this.setupVisualizationState({
    hideText: ["#intro-text", "#presentation-text"],
    styleType: "zone",
    tooltipType: "zone",
  });
}

The state machine is calling the setupVisualizationState() method, which is responsible for configuring the various aspects of the visualization –like hiding and showing text, updating the styles and tool-tips, and so on. This way, each step is building upon the last, creating therefore a seamless and engaging user experience.

And as I delved deeper into the concept, I couldn't help but marvel at the elegance of it all. The way it handled the complex interplay of user interactions, map layers, and visualization settings – it was like watching a master puppeteer, pulling the strings with effortless grace.

But what really struck me was the way the state machine anticipated the user's every move. It was as if it could peer into the future, anticipating the next button click or menu selection, and adjust its course accordingly. Have a peek at the handleStepChange() method:

/**
 * Handles Next/Previous buttons change.
 *
 * Handles the step change when the "Next" or "Previous" button is clicked.
 * @param {number} direction - The direction of the step change (1 for next, -1 for previous).
 */
handleStepChange(direction) {
  const newStep = this.state.currentStep + direction;
  if (newStep >= 0 && newStep <= 10) {
    this.state.currentStep = newStep;
    this.updateVisualization(newStep);
    this.updateButtonState();
  }
}

Here, the method is carefully monitoring the user's button clicks, adjusting the current step, updating the visualization to the corresponding step, and of course updating the state of the buttons themselves. We wouldn't like to get lost along the way, right?

So when the user makes a move, the state machine springs into action, smoothly transitioning from one state to the next, updating the visualization with precision and finesse.

And by watching in motion this humble piece of code, I saw the embodiment of a fundamental truth: the most complex systems can be tamed, if only we have the patience and the vision to understand them.

Enhancing the visualization

Now, this visualization tool we've been tinkering with - since we aim to elevate the viewer's experience and make it a veritable dance of data and pixels, we'll need to furnish it with neat little goodies. Let's review some:

/**
 * Adds circle markers to the map representing the population percentage of each GeoJSON feature.
 */
addPopulationDots() {
  this.state.geoJsonLayer.eachLayer((layer) => {
    const { feature } = layer;
    const radius = Math.max(feature.properties.pop_percentage * 5);
    const dot = L.circleMarker(layer.getBounds().getCenter(), {
      radius,
      color: config.COLORS.DEFAULT,
      weight: 1,
      opacity: config.OPACITY.DEFAULT,
      fillOpacity: config.OPACITY.LOWER,
      fillColor: config.COLORS.DEFAULT,
      zIndexOffset: 1000 - radius,
    }).bindTooltip(
      `${feature.properties.nom_complet}<br>Part de la pop. rég. ${feature.properties.pop_percentage}%`,
      { permanent: false, direction: "center", className: "custom-tooltip" }
    );
    dot.addTo(this.state.map);
  });
}

Take, for instance, the way it handles the population visualization in the snippet provided above. The addPopulationDots() method adds those little dots to the map, each one a representation of the humanity in a visual symphony of our fellow citizens. And wait.. with a flick of the wrist, the whole thing can be cleared away leaving the canvas clean and pristine, ready for the next movement.

But it doesn't stop there, oh no! This state machine, it's got tricks up its sleeve. Look below at the way it manages the styles and tool-tips, seamlessly transitioning from one map mode to the next. Each one is a unique expression of the data, and therefore a different glimpse through a lens which to view the world.

/**
 * Generates a style for the GeoJSON layer .
 *
 * Returns a style object based on the specified feature, type and option.
 * @param {object} feature - The feature to generate the style for.
 * @param {string} type - The type of style to generate. (e.g., "default", "zone", "consumption", "production", "ratioEnr").
 * @param {object} options - Optional options for the style.
 * @returns {object} The generated style object.
 */
generateStyle(feature, type, options = {}) {
  const styleMap = {
    // ... various style generation functions
  };

  return (styleMap[type] || styleMap.default)();
}

Now look at the way it handles the transitions between states. Like a skilled dancer, the state machine glides effortlessly from one step to the next, each move carefully choreographed to create a seamless and captivating performance.

/**
 * Updates the visualization based on the specified category and type.
 * @param {string} category - The category of the visualization (e.g., "consumption", "production", "final").
 * @param {string|Object} type - The type of visualization within the category.
 */
updateVisualizationByType(category, type) {
  const options = {};
  // ... set up options based on category and type
  this.updateGeoJsonStyle(category === "final" ? type : category, options);
  this.updateTooltips(category === "final" ? type : category, options);
}

And when the user wants to explore different aspects of the data within an interactive menu, the state machine is there to guide them, offering up a display of options that integrate with the overall visualization.

/**
 * Shows a radio button menu on the map with the specified types and name.
 * @param {Array} types - An array of objects representing the menu options, each with an id and label.
 * @param {string} name - The name of the menu.
 */
showMenu(types, name) {
  this.hideMenu();
  $(config.SELECTORS.MAP).append(this.createMenu(types, name));
  $(`input[name="${name}-type"]`).on("change", (e) =>
    this.updateVisualizationByType(name, e.target.value)
  );
}

That's all folks, we did it, and I can tell that the visualization in motion you see, is a delicate pas de deux between a human and a machine, each one responding to the other's every move. In the end, what do we have? A visualization that doesn't just display data, but rather, one that makes it sing by telling a story, just like a telltale that captivates and inspires people.

The complete source code is available on this github repository.

(Cover picture: Modern Times, 1936).

Basic Selenium – Bonus

brk — Thu, 20 Feb 2025 15:36:06 +0000

The tome of wise practices

Some tips on improving code readability.

The Code header

Where the purpose, the author, and the date of creation are inscribed

The header, ahh.. the header. That little snippet of code at the top of the script is like a preamble, a clearing of the throat before the real business begins. What purpose does it serve? Well, the header is a kind of introduction, letting the reader know what's to come with some basic information.

A main() function

Let the Code be governed by a singular, mighty function

Ah, the main() function - now there's a topic! It's like the heart and soul of your code, the place where everything comes together. See, when you're working on a code project, it's like you're building a big machine, right? And the main() function, well, that's the control panel, the place where you pull all the levers and push all the buttons to make the whole thing rock. In order to use that machine, you gotta follow a certain set of steps right? Turn the key, check the fuel, prime the pump.. Well, that's what the main() function is all about - it's the sequence of events that gets your code up and running.

The parameters, guardians of function

Each one a gatekeeper, ensuring the proper flow of information.

Parameters are those little vessels that carry the lifeblood of our functions.
Classes or variables?
Now, I know what you're thinkin' - "But George, how do I know which one to use?" Well, my friend, it all comes down to the task at hand.
If you're dealing with a relatively simple setup like here, them variables-as-parameters might be just the ticket you need. But if you're working on somethin' a little more complex, with all sorts of moving parts and interdependencies, well, them classes, they're the way to go. Classes, are the real MVPs of the bunch, and they will help keeping everything organized and running smooth as silk.

The docstrings, the illuminating scrolls

Where the function's purpose, its workings, and its returns are documented.

Alright, people, let me tell you about these things called docstrings. Now, I know what you're thinking - "Docstrings? What in the Sam Hill are those?" Let me break it down for you.
These docstrings, they're like little notes, little snippets of information that you tack on to your code, just to give folks a heads up on what's going on. It's like when you're doing some work around the house, and you leave a little note for the neighbour, just to let ‘em know what you're up to.
Now, these docstrings, they come in all shapes and sizes. But the way I see it, these docstrings, they're not just about the code, no sir. They're about the people too. So, when you write one of these things, you're not just explaining what the code does, you're telling a story. You're givin' people a little glimpse into your mind and your thought process. And this makes t-h-e difference because on bigger projects, we all know you will never code alone.

Type hints as vigilant sentinels

Ensuring the integrity of the data.

Listen up, I'm about to let you in on a little secret when it comes to this Python business. It's all about them type hints, they're guardrails that keep you from taking a wrong turn by telling your fellow programmers, "hey, this is what I'm expecting here, so don't go messing it up, y'hear?"
See, these type hints, they're some kind of guardians of your program, keeping a watchful eye to make sure you don't go trying to mix apples and oranges, so to speak. When you're dealing with a language as flexible as Python, that's a darn good thing to have in your corner. The more you use ‘em, the more you'll see just how powerful they can be.

If you skipped the project introduction, feel free to check the first chapter (setup), the Code overview, or just dive deep into the source code in the previous chapter!

The code is available on Github.

(Cover picture: unknown).

Basic Selenium – The Easy Peasy Introduction, Chapter 3 of 3

brk — Thu, 20 Feb 2025 15:28:52 +0000

Automating web-based tasks with Selenium? Efficiently. That's the name of the game here, so.. Take the reins and make technology work for you.

A coding story in three chapters (with a bonus).

The Odyssey of the Code
- I. In the realm of the browser, a puppet is forged
- II. The opening of the file
- III. Sentence splitting action unleashed
- IV. The forging of the chunks
- V. Print statements illuminate the path
- VI. The sorcery of the selectors
- VII. The bridge of translation is crossed
- VIII. The file is written
Conclusion

The Odyssey of the Code

The Taming of the Firefox

I. In the realm of the browser, a puppet is forged

Where the mighty browser is bound to our will, as if by sorcery.

def initialize_browser() -> webdriver.Remote:

    driver_service = Service(GECKODRIVER_PATH, log_output="geckodriver.log")
    options = webdriver.FirefoxOptions()
    options.binary_location = FIREFOX_PATH
    if HEADLESS:
        options.add_argument("-headless")

    return webdriver.Firefox(service=driver_service, options=options)

First up, we've got this driver_service business. Now, I know what you're thinking - "George, what in the world is a driver service?" Well, let me tell you, it's the piece of code that's gonna make this whole Selenium thing work. It's like a bridge between our script and the middleman, I name the geckodriver! And after that service is up, the gecko’s gonna turn our Firefox into a nice marionette.
And where do we get this driver service, you ask? From the GECKODRIVER_PATH, of course! That's the path to the executable that's gonna make Firefox dance to our tune. Do you know what else it's gonna do? It's gonna log everything that's happening, right into the geckodriver.log file. That way, if anything goes sideways, we can take a peek under the hood and see why’s that.
Now, the options. See, we're creating a shiny new set of FirefoxOptions, and we're gonna deck them out with some goodies. First up, we're pointing it to the FIREFOX_PATH, making sure we've got the right browser to work with.
And then, if that HEADLESS variable is set to True, we're gonna add a little -headless argument to the mix. That means the browser's gonna run without a visible window, stealthier than your average ninja. No need for all the bells and whistles (and windows), we're just here to get the job done, right?
And finally, we're wrapping it all up by returning a brand-spankin' new instance of the Firefox webdriver, complete with our custom service and options. This one's like a well-oiled machine, and we're the ones behind the wheel.
Just remember: if you ever find yourself scratching your head, wondering what in the world is going on, just take a peek at that geckodriver.log file. It's like a crystal ball, and it might tell you everything you need to know.

II. The opening of the file

Like an ancient tome unveiling its secrets.

def load_input_file(file_name: str) -> str:

    try:
        with open(file_name, "r", encoding="utf-8") as f:
            return f.read()
    except FileNotFoundError:
        print(f"File {file_name} not found. Please check, and try again.")
        raise

Alright, folks, gather 'round, because we're about to dive into some more of this code. Now, this right here, this is the kind of stuff that separates the wheat from the chaff.
First up, we've got this try block. Now, I know what you're thinking: "George, what in the world is a 'try' block?" Well, let me tell you, it's the statement that’s gonna keep us from running headfirst into a brick wall every time something goes wrong. And trust me, in the universe of programming, something's always going wrong.
See, what we're doing here is we're trying to open a file, right? And we're using this fancy-schmancy open() function to do it. And that's where the try block comes in.
If everything goes according to plan, and the file is there, waiting for us with open arms, we're gonna read the contents and hand 'em back, no problem. "But George, what could possibly go wrong?" Well, my friends, the world is a cruel and unpredictable place (especially when you’re learning software development), and sometimes, those files, they just up and disappear. But don't you worry, we've got a plan and that's where the except block comes in.
If that 'FileNotFoundError' rears its ugly head, we're gonna print out a nice, friendly message, letting the user know that the file they were looking for is.. not there..
It's like a safety net, a way to keep the wheels from falling off even when the road gets a little bumpy. And you know what they say, "the more you plan for the unexpected, the less unexpected it becomes." Or something like that.. I don't know, I'm just making it up as I go along.

Anyway, the point is when life hands you lemons, you make lemonade. And when life hands you FileNotFoundError's, you just smile ;)

III. Sentence splitting action unleashed

The fellowship of words is broken.

def split_text_into_sentences(text: str) -> list:

    return re.split(r"(?<=[\.?!])", text)

Okay, listen up, because we're about to dive into some serious sentence-splitting action.
Now, you might be looking at this line of code and thinking: "what in the world is going on here?" Well, let me tell you, it's a thing of beauty: we're taking this text that we've been handed, and we're gonna break it down into its individual sentences.
And how are we doing it, you ask? With the power of regular expressions (regex) and its re.split() implementation, that's how. "But George, what's a regular expression?" Well, my friends, it's a language all its own, a way of describing patterns in ways that would make your head spin. You can tinker with it here.
But don't worry, we're not gonna get too deep into the weeds here. All you need to know is that this little regular expression is the key to our success. It's gonna look for those periods, question marks, and exclamation marks, and it's gonna use them as the boundaries to split our text into individual sentences. But careful, this regex has its limits, see, every language is like a delicate little dance, with each word and phrase movin' in perfect harmony. Some Pros, they’re like aware of that more than us so they built up nice little tools using carefully crafted and more accurate natural language processing (NLP) formulas (for example you can experiment with the nltk library). For the sake of simplicity, let’s stick to the basics with the regex way.

The next time you find yourself staring at a wall of text, wondering how in the world you're gonna make sense of it all, just remember this little line of code and no more trying to figure out where one sentence ends and the next one begins. And who knows, maybe one day, you'll be the one writing the regular expressions turning chaos of strings into order.

IV. The forging of the chunks

The awakening of mighty blocks of text.

def generate_chunks(text: list, char_limit: int = CHAR_LIMIT) -> Generator[str, None, None]:

    chunk = deque()
    chunk_length = 0

    for sentence in text:
        sentence_length = len(sentence) + 1  # Add 1 for the space character
        if chunk_length + sentence_length > char_limit:
            yield " ".join(chunk)
            chunk.clear()
            chunk.append(sentence)
            chunk_length = sentence_length
        else:
            chunk.append(sentence)
            chunk_length += sentence_length

    yield " ".join(chunk) if chunk else ""

Hmm, we're about to dive into some serious text-chunking action now. You see, we've got this text that we need to translate, but we can't just send the whole thing off to the translation service all at once. Nah, that would be way too easy. Instead, we've gotta break it down into manageable chunks, little bite-sized pieces that the service can handle without breaking a sweat. That's where this generate_chunks() function comes in handy. It's like a master chef, carefully slicing and dicing the text, making sure each piece is the perfect size.
First, we set up a little deque, a fancy data structure that's gonna help us keeping track of the current chunk.

A deque, that's just a chic way of saying double-ended queue, obviously we deal with small amounts of data here but I wanted to give a try to this exotic thing. Your less sophisticated arrays would work fine there too. Just remember that usual pop and append methods don’t perform fast on items on the opposite side of the line. So Python’s collections module provides that class called deque that’s specially designed to provide fast and memory-efficient ways to append and pop item from both ends.

And then, we start looping through the sentences of the "text" block (we provided it as an argument when we called the function), and for each sentence, we're gonna figure out its length, -including the space character at the end.
Now, you might be wondering, "but George, how do you know when to start a new chunk?" Well, my friends, it's all about that char_limit variable and the one called chunk_length that we use as a counter to keep a running tally of the length of the current chunk. If the current sentence is gonna fit within the chunk size character limit, well, we're just gonna add it on and update the chunk length accordingly. That’s the normal way to go. Got it?
Now, if the length of the current sentence, plus the length of the chunk we've got so far, pushes us over the character limit we've set, well, we're gonna do a few things. First, we're gonna take all the sentences in the chunk and join 'em up with spaces, and then we're gonna yield that beasty. That just means we're gonna spit it out and move on to a new chunk.
Moving on to the next chunk means we're gonna clear out the chunk variable and start fresh, adding the current sentence to it and resetting the chunk length to just the length of that sentence, and then, just like before, we’ll keep rolling until we hit the limit again.
Finally, after we've gone through all the sentences, we're gonna yield the last chunk, if there is one.

That’s it for our symphony of text-chunking perfection. And who knows.. maybe you'll even find yourself singing the praises of them deques and character limits soon.

V. Print statements illuminate the path

The chronicles of progress are Written.

def print_preprocess_infos(input_text: str, chunks: List[str]) -> None:

    print("------------------")
    # Delete extra spaces gathered during generate_chunks
    input_text_with_spaces = input_text.replace(
        ".", ". ").replace("?", "? ").replace("!", "! ")
    print(
        f"Input text contains {len(input_text_with_spaces)-(len(chunks)-1)} characters.")
    print("------------------")
    print(f"Found {len(chunks)} chunks!")
    print(f"Sizes: {[len(s) for s in chunks]} characters each.")
    print("------------------")

Alright, but George, why do we need all these print statements? Isn't the script just supposed to, you know, just work?"
Well see, we've got this script that's doing all sorts of stuff, translating text and whatnot. But how are we supposed to know if it's working, huh? That's where these print statements come in, my friends, they're like the canaries in the coal mine. They're the early warning system that's gonna deal us some intel to let us know something's gone awry.
We're gonna print out the length of the input text. Because let's be real, how are we supposed to know if this thing is working if we don't know how much text we're dealing with, am I right? It's like trying to bake a cake without knowing how much flour you've got. There's more.. We're also gonna print out the number of chunks that the script has generated. And to top it all off, we're gonna print out the length of each of those chunks. Because, honestly, who doesn't love a good old-fashioned character count? It's like a little treasure hunt, but instead of finding gold, we're finding out how many letters are in each sentence.
A little apparté for when you will be contributing to bigger projects: there is that other breed of statements called loggers. They're the quiet ones, they slip in the back door and get the job done without all the fanfare of the print. Just keep in mind they're the ones that keep a careful record when things start to get a little hairy.

VI. The sorcery of the selectors

Their spell cast the way.

def get_input_textarea_element(driver: webdriver.Remote) -> WebElement:

    return WebDriverWait(driver, TIMEOUT).until(EC.element_to_be_clickable((By.CSS_SELECTOR, INPUT_AREA)))

Now it’s time to jump into some serious Selenium wizardry.

The input field our marionette will search for.
You see, we've got this input field that we need to find on the web page, and we can't just go barging in there, hoping it'll be there. Instead, we've gotta harness the power of Selenium, which is gonna help us navigate this digital bytescape.
To understand this whole pasta here, We need to know more about’em selectors. See, when you're coding a website, just like a building, you got all sorts of elements fitting together - your headings, your paragraphs, buttons, an' the like. Each one of those elements, it's got its own little personality, just like a set of traits and characteristics. And to identify them, you can use all sorts of different selectors: element selectors, class selectors, ID selectors, and many more other lads. Find out more here.

That's where this get_input_textarea_element() function comes in. It's like a secret agent, carefully scanning the page, looking for that elusive input field where you insert the text you want to translate.
But there’s a challenge to it: what's the point of finding the input field if it's not even ready for us to use? So first, we create a WebDriverWait object, and we instruct it to wait for up to TIMEOUT seconds (we've set it at the beginning as a parameter, by the way) for the element to be loaded and clickable thanks to this intriguing EC.element_to_be_clickable() thing that tells Selenium exactly what we're looking for. In this case, we're saying, "Hey, Selenium, find me an element that's clickable, and it's gotta match this CSS_SELECTOR. But.. how did I found that damn selector we've got here? Well, invoke the inspector by hitting F12 in Firefox. Then try the ctrl+shift+C combo, point & clic your target and in the inspector you'll be able to right clic the highlighted piece of code and extract its info.

Concretely we’re looking to mimic human behavior just like when you paste your first chunk of text. Let’s break it down again:

You select the input field by clicking on it (it’s selected).
Depending of your laziness level, either you start typing or you just paste some text into that area.

Well, our driver should be instructed to do the same.
Now, I know what you're thinking, "but George, what if the element never becomes clickable? What then?" Well, friends, that's where the WebDriverWait comes to the rescue. If the element doesn't become clickable within the TIMEOUT period, the whole thing is gonna throw an exception, and we'll know that something's gone wrong. Maybe the website has been updated and our old CSS selector pal is no more, or a networking problem occured somewhere. Whatever.
If everything goes according to plan, and Selenium manages to find that input field and confirm that it's ready for us to use, well, that's when the wonder happens. We're gonna return that element, and the rest of the script is gonna be able to work its magic.

VII. The bridge of translation is crossed

Meaning is conveyed across the void.

def translate_text(driver: webdriver.Remote, input_field: WebElement, chunks: list) -> list:

    translation = []

    for index, chunk in enumerate(chunks):
        print(f"Translating chunk {index+1} of {len(chunks)}")
        print("------------------")
        input_field.send_keys(chunk)
        print("Fetching translation...")
        time.sleep(SLEEP_TIME)

        translated_chunk = driver.find_element(
            By.CSS_SELECTOR, OUTPUT_AREA).text

        translation.append(str(translated_chunk))
        input_field.clear()

    return translation

Alright, friends, gather 'round, because we're about to witness the main event, the grand finale and the moment you've all been waiting for - the translation process! Now, I know what you're thinking, "but George, how in the world is this script gonna take all those chunks of text and turn them into a beautiful translation?" Well, my friends, watch out.

First, we're gonna set up an empty list called translation. Our little treasure trove of linguistic gold. This is where we're gonna store all the translated chunks.
And then, we're gonna start looping through those chunks, one by one. Now, I know what you're thinking, "but George, how are we gonna keep track of which chunk we're on?" Well, that's where the enumerate() function comes in, my friends. It's gonna give us the index of each chunk, so we can keep tabs on our progress.
As we loop through each chunk, we're gonna print out a little separator, just to let the user know that we're hard at work. And then, we're gonna send that chunk of text to the input field, using the send_keys() method. It's like we're typing the text over to the translation service, saying, "Here, take a look at this! A new text chunk for you to translate."
But we can't just sit back and wait, oh no, that would be way too easy. Instead, we're gonna print out a little message, letting the user know that we're fetching the translation. And then, we're gonna hit the time.sleep() function, giving the translation service some time to achieve its task.
When the time is up, we're gonna use Selenium again to find the output area on the web page, and we're gonna grab the text that's been translated by now. We're talking about pure algorithmic alchemy, folks, turning one language into another with the click of a button. The thing there is just that we’re not behind the wheels anymore.
So once we've put our hands on that translated chunk, we're gonna append it to our translation list. And then, just to be on the safe side, we're gonna clear out the input field, making sure we're ready for the next chunk.
When we've gone through all the chunks, and collected all those translated gems, we're gonna return the whole (translation) list which is the result of all the translations we’ve collected.

VIII. The file is written

The chronicle of the realm is inscribed.

def write_output_file(file_name: str, translation: list) -> None:

    with open(file_name, "w", encoding="utf-8") as f:
        f.writelines(translation)

The last step starts by calling write_output_file(), and let me tell you, it's one of the unsung hero of this whole operation. Because, let's be honest, what's the point of all this translation process if we can't actually, you know, save the results somewhere?
So, here's how it works. First, we're gonna open up a file, using that with statement. And let me tell you, that with statement, it's like a handshake that tells Python, "Hey, I'm about to do some serious file-handling business, so don't you dare interrupt me!" And what are we gonna do with this file, you ask? Well, my friends, we're gonna write some text to it. But not just any text, oh no, we're talking about the fruits of our labor, the translated chunks that we've been slaving over for who knows how long.
Now, I know what you're thinking, "but George, how are we gonna get all those chunks into the file?" Well, that's where the f.writelines() function comes in. It's like a magic wand, taking our list of translated chunks and turning them into a cohesive piece of writing. When that file finally gets saved, it's gonna be all those chunks of text, neatly packaged up into a tangible reality: a file!
But you know, it's not just about the end result, folks. It's about the journey, the process of getting there. And this write_output_file() function, was the final step in that journey. A cherry on top, mic drop moment that says, "We did it, folks, we translated the heck out of that text!"
kudos

Conclusion

We’ve been:

Taming that Selenium machine (installation and configuration)
Managing files (loading a source file and writing results into another one).
Witnessing that the true sorcery of this tutorial lies in the automated process of translating a text by splitting it into sentences, stacking them into piles of chunks sent one by one to the online translation service, through Selenium, without ever exceeding a character limit!
And the goodies: raise a glass to the coder's toolkit, by implementing some good practices (using a header, organizing the code using functions, parameters and a main(), use docstrings and type hints).

That's just the beginning, isn't it? This Selenium business, it's got so much more to offer. The possibilities, they're practically endless, my friends. So I want you to take this code, tinker with it, experiment, see what else you can make it do. You’re ready to go.
Try it out on different translation services, see how it handles the variations. Heck, see if you can make it do your taxes while you're at it (okay, maybe not that, but you get the idea). The point is, this is tip of the iceberg. So, what are you waiting for? Good luck!

The code is available on Github.

(Cover picture: Cat People, 1942).

Basic Selenium – The Easy Peasy Introduction, Chapter 2 of 3

brk — Thu, 20 Feb 2025 15:17:29 +0000

Automating web-based tasks with Selenium? Efficiently. That's the name of the game here, so.. Take the reins and make technology work for you.

A coding story in three chapters (with a bonus).

The Fellowship of the Functions

A Functional Overview:
There's a whole lot going on under the hood, and you're gonna want to know what makes this thing tick.

For the newcomers out there, on using functions:
Sometimes you got bits of code that you need to use over and over again, right? Well, functions, they make your code a whole lot easier to read and understand. I mean, think about it: instead of trying to decipher a tangled web of spaghetti code, you got these nice, neat little packages that do one thing and do it well. And when you need to make a change, well, you just gotta tweak the function, and b-a-m, problem solved. No muss, no fuss.

Word of advice: Embrace 'em, you love 'em. Trust me, your future self is gonna thank you for it.

First, let's have a look at this simplified version of the main function:

# Main Code
def main():

    # Initiate an instance of the web browser
    driver = initialize_browser()
    driver.get(LINK)

    # Wait a bit
    time.sleep(SLEEP_TIME)

    # Parse the text
    input_text = load_input_file(INPUT_FILE)
    sentences = split_text_into_sentences(input_text)
    chunks = list(generate_chunks(sentences))

    # Eventually print some more details 
    if PRINT_DETAILS:
        print_preprocess_infos(input_text, chunks)

    # Find the input field
    input_field = get_input_textarea_element(driver)

    # Translate the text
    translation = translate_text(driver, input_field, chunks)

    # Save to output file
    write_output_file(OUTPUT_FILE, translation)

    # Shut down the browser instance
    driver.close()

The Initialization of the Browser

In which the Selenium-forged steed is summoned, and the journey begins.

First, we've got the initialize_browser() function. This is where the magic starts - it's setting up a brand new instance of the Firefox browser, all decked out with our custom options. Headless mode is on by default! No need for a window to appear, we're going full stealth mode here.

The Parsing of the Text

Where the words are divided into manageable chunks, as if by the wisdom of the regex.

Next, we've got load_input_file(). This one's pretty straightforward - it's just reading the contents of a file and handing us back the text.
Then there's split_text_into_sentences(). This is where the script takes that input text and breaks it down into individual sentences. Gotta make sure we're not overwhelming the translation service, you know? Bite-sized chunks are the way to go.
And speaking of those chunks, that's where generate_chunks() comes in. It's taking those sentences and slicing them, making sure each sentence-block is small enough to play nice with the translation service. No more hitting character limits.

The Gathering of the Fields

In which the input field is sought and found.

Now, the real showstoppers: get_input_textarea_element(). This is the function that use Selenium to find the right spot on the web page to do our work. I mean the input field, where we're gonna pour in our text. Without it, no circus troupe at your fingertips, ready to leap through hoops an' do backflips at your every whim.

The Fetching of the Results

The final step, where the fruits of the labors are harvested and the story ends.

Finally, we've got translate_text() and write_output_file(). These are the heavy hitters. translate_text() is where the rubber meets the road, sending those sentence chunks off to be translated and bringing back the long awaited goods. write_output_file() is the grand finale, putting the whole shebang down on paper (or, you know, in a file).
Whew, that's a lot to take in, I know. But well, once you've got this thing up and running, you’ll see, it's gonna be smooth sailing. Just sit back, let the wind.. Hmm.. the script do its thing.

If you skipped the project introduction, feel free to check the first chapter or just dive deep into the source code in the next chapter!

The code is available on Github.

(Cover picture: Laura, 1944).

Basic Selenium – The Easy Peasy Introduction, Chapter 1 of 3

brk — Thu, 20 Feb 2025 15:08:34 +0000

Automating web-based tasks with Selenium? Efficiently. That's the name of the game here, so.. Take the reins and make technology work for you.

A coding story in three chapters (with a bonus).

About the Project

There I was, sitting on a chair, having to gather all these newspaper translations, from who knows where, in who knows what languages. Let me tell you, it was a real slog. The c h a r a c t e r l i m i t, just wouldn't let me get the full show. Nothing but snippets, snippets, snippets...

Then, the idea struck. This script, see, it's gonna let you bypass all that. No more copying and pasting. Just grab a coffee, make it strong, black and wait for the whole enchilada - right there in front of you. A real time-saver, I'm telling you.

The best part? Turns out, it wasn't just about solving an immediate problem: it was about deepening my know-how, getting my hands dirty with some real-world programming. A win-win, if you ask me.

Installation and Configuration

Introduction

Alright, let's crack open this code. Step by step, we're gonna peel back the layers of that machine where each piece is fitting in like a puzzle. By the time we're done, you're gonna have a whole new appreciation for Selenium automation and what it can do.

And beware, you're not just gonna be learning for learning's sake. Nah, this is about equipping you with the tools to tackle your own challenges. Whatever got you tied up in knots, this is your chance to untangle it and make it sing.

Installation

"Time to get this thing installed and ready to rock."

First things first - you're gonna need to clone or download the source code. The repository is online here. Don't worry, I'll wait.

If you don't already have Firefox on your system, you're gonna need to get that taken care of. For Ubuntu and other Debianistas folks out there, it's a simple enough fix:

sudo apt update && sudo apt install firefox

It’s also possible to use Chrome instead of Firefox.. It’s not covered here so if you really want it that way, it’s this way there.

Next, we've got some packages to install. Nothing too crazy, but we gotta make sure we've got all the tools ready to roll. So fire up that trusty old pip and run:

pip install -r requirements.txt

Boom. Done and done.
Now, the real kicker - the GeckoDriver. This is the secret sauce that sits behind the scenes, translating your commands into a language web browsers can understand.
You're gonna want to head over to the Mozilla repository, grab the latest version, and get it all set up. I'm talking extraction, permissions, symbolic links - the whole nine yards. And you know the magic? There’s a bash spell ready out there for that:

wget https://github.com/mozilla/geckodriver/releases/download/v0.35.0/geckodriver-v0.35.0-linux64.tar.gz -O /tmp/geckodriver.tar.gz \
&& sudo tar -C /opt -xzf /tmp/geckodriver.tar.gz \
&& sudo chmod 755 /opt/geckodriver \
&& sudo ln -fs /opt/geckodriver /usr/bin/geckodriver \
&& sudo ln -fs /opt/geckodriver /usr/local/bin/geckodriver

That's a mouthful, I know. But trust me, it's worth it. Once you've got all that squared away, you're gonna be almost ready to start automating like never before.

Configuration

Alright, folks, time to dive into the nitty-gritty of this configuration stuff. Because let me tell you, if you don't get this part right, the whole thing's gonna be about as useful as.. a lawn mower in the middle of the Mojave.
First up, we've got those FIREFOX_PATH and GECKODRIVER_PATH variables. Now, I know what you're thinking - "But how in the world am I supposed to know where these things are hiding on my system?" Well, fellas, depending on the one you’re looking for, there's a little bash command you can run to ferret that out:

which firefox

About the Gecko.. Remember that installation process we went through earlier? Well, the path you set up there is the one you'll want to use here. Not sure about it? No problem, same sauce, Just run:

which geckodriver

And Zap! Paste that info into the right place. Easy, right?
Now, the HEADLESS parameter - this one's a bit of a wild card. See, you can run this whole thing in headless mode, which means the browser will run and do its job but won't actually show up on your screen. Kinda like a ninja, you know? But if you want to see what's going on, and watch your Firefox acting like possessed by a spirit you can set it to False and watch the magic unfold – try and see.

As for the SOURCE_LANG and OUTPUT_LANG, well, pretty self-explanatory. Just pick the languages you want to translate to (source) and from (output), and the script will handle the rest. If you're looking to learn a little more about the available languages, take a glance at that readme file in the repository here.
Last but not least, we've got the CHAR_LIMIT, TIMEOUT, and SLEEP_TIME. These are all about fine-tuning the performance. TIMEOUT, and SLEEP_TIME depend mostly on the speed of your network connection. It’s about how long we oughta wait for the website to be fully loaded so we can proceed with the translation. CHAR_LIMIT is linked to deepL’s restrictions (currently 1500 characters). Tinker with them as needed, but be careful - you don't want to break the whole machine.
The INPUT_FILE and OUTPUT_FILE - those are the ones you'll want to point to your own input and output files. Simple enough, right? Just make sure you're pointing the input to something that actually contains text, because that's what this script is built to handle.
Now, a quick word of warning – the script is not perfect - if the output file doesn't exist yet, the script's gonna go ahead and create it for you. But if it does exist, well, buckle up, because the script's gonna overwrite it every time you run it. So, you know, maybe keep an eye on that (and specify a different output each time) if you don’t want to see your previous achievements disappear..

Wait.. I almost forgot.. You're a WSL User? WSL now supports running Linux GUI applications in a fully integrated experience. On older configurations, you might need this if you intend to run Firefox as a marionette with its GUI.

Usage

Once you've got those parameters all squared away, fire up the terminal, go to the script's folder and just run:

python translate.py

Ka-boom, the show’s hit the road. The script's gonna take that input file, work its translation-y wonders, and spit out the results into your output file. Fear not, my friend just keep an eye on that output file, and you'll be able to see the fruits of that labor, plain as day.
What are you waiting for? Get out there, update those parameters, and let's see what kind of translation magic you can work!

If you're curious about the code you can get a quick overview in the second chapter or just dive deep into it (third chapter)!

The code is available on Github.

(Cover picture: Double Indemnity, 1944).

Forem: brk

Oh, the water, the water!

The data is out, the challenge has started..

The Curious Case of the Visualization State Machine

A never-ending mechanic

Enhancing the visualization

Basic Selenium – Bonus

The tome of wise practices

The Code header

A main() function

The parameters, guardians of function

The docstrings, the illuminating scrolls

Type hints as vigilant sentinels

Basic Selenium – The Easy Peasy Introduction, Chapter 3 of 3

Contents

The Odyssey of the Code

I. In the realm of the browser, a puppet is forged

II. The opening of the file

III. Sentence splitting action unleashed

IV. The forging of the chunks

V. Print statements illuminate the path

VI. The sorcery of the selectors

VII. The bridge of translation is crossed

VIII. The file is written

Conclusion

Basic Selenium – The Easy Peasy Introduction, Chapter 2 of 3

The Fellowship of the Functions

The Initialization of the Browser

The Parsing of the Text

The Gathering of the Fields

The Fetching of the Results

Basic Selenium – The Easy Peasy Introduction, Chapter 1 of 3

About the Project

Installation and Configuration

Introduction

Installation

Configuration

Usage