You might be wondering what’s new in Bactopia? And the answer is a lot, but also… not a lot, but in a good way! How can that be? Well, about that…

Bactopia is all Strict, Static and Records now!​

Over the last year Nextflow (the engine that Bactopia is based on) has undergone a significant metamorphosis, with the introduction of Strict Syntax and Static Types in v25, and now Record Types in v26. Haha if you’ve ever looked at the Bactopia code base, you would have quickly seen that it was the least strict thing. Bactopia took advantage of many things in Nextflow that weren’t really the intended purpose (e.g. launching 60+ workflows from a single workflow…).

So, to answer your first question: how did Bactopia change a lot, without changing a lot?

Well, in order to implement strict syntax, static types, and record types, I had to essentially rewrite the code base, almost from scratch (which again is not necessarily a bad thing!). As you can see in the image below, v4 of Bactopia changed 2,641 files, with 140k additional lines of code and 55k removed lines of code. It's true! Check out the CHANGELOG!

In version 4 of Bactopia, I’m happy to report that I’m now compliant with the future of Nextflow! Which as a side-effect, means Bactopia should now play much nicer both locally and in the cloud.

While Strict Syntax, Static Types, and Record Types aren’t exactly “required”, I think their adoption has put Bactopia in a much better place than it was before. Cheers to the folks at Seqera (who maintain Nextflow) for getting that implemented!

Co-development with AI​

[can you write a message about the usage of llms in Bactopia] Haha, just kidding, it's still me writing this post! But, on a more serious note, going forward Bactopia will be utilizing AI in the future developments and maintenance. I’m the only developer and maintainer of Bactopia, so I have to balance what I can, to avoid burnout going forward. However, my usage of AI is not without a lot of thought. There won’t be any AI agents getting unrestricted access to do what it wants with Bactopia. Trust me, Bactopia is something I care about deeply, and I plan to continue maintaining ownership of it.

I’ve made extensive efforts to ensure there are specific guardrails for AI to follow, as well as ways for me to verify those guardrails are being followed. With the introduction of strict syntax, static types, and record types, I was able to incorporate numerous patterns that are well suited to LLMs and these guardrails I’ve set up. At the end of the day, I will still be very much involved in reviewing any changes made by LLMs and pushing Bactopia forward. I just wanted to briefly state my stance on the usage of AI and LLMs to help co-develop Bactopia.

I feel a more lengthy post is still needed to demonstrate my process when using LLMs for Bactopia. I’ll be sure to get that out soon!

Bactopia is easier to support now​

With that out of the way, I mentioned I spent a lot of time rewriting Bactopia, and turns out the rewrite didn’t move the dial much, but that’s a good thing. You might be wondering, how is all this effort to end up where you started a good thing?!? Well, when you get the opportunity to rewrite something, you get to rewrite it based on your current experience and abilities. I’m not the same bioinformatician I was over 7 years ago, when Tim and I first released Bactopia. I sometimes wonder: 'why did 2017 Robert do it this way?' and 'why is 2026 Robert so much more gray?!'

But, to me at least, this rewrite provided the opportunity to unload a lot of technical debt past Robert built up over the years. I was able to standardize many pieces of Bactopia, so a module is a module is a module is a module, etc…. I also got to move a lot of the non-Nextflow pieces out of the Bactopia Nextflow pipeline and into bactopia-py and nf-bactopia.

While pytest still worked, I was able to transition to nf-test for all Nextflow files, not just the subworkflows. With this transition, at the moment, Bactopia now includes 246 tests, which test everything (modules, subworkflows, and workflows) using real data from the revamped bactopia-tests repo.

I was also able to make the documentation built from of in-line GroovyDoc. Not only that, I moved from MkDocs Material to Docusaurus, and even got this new fancy address! I personally think it will be much easier to maintain, and am very curious to get your opinions and feedback!

All this, to say:

If you are a user of Bactopia, what you need to take away is, I now find Bactopia much easier to build and maintain. Tim and I are in the kitchen cooking up some fun things for the future of Bactopia. So keep an eye out!

P.S. As with all major releases, there will be bumps in the road, and I’m sure I may have missed something. If you find bugs or a feature I missed, just let me know! Maybe even find me on the Bactopia Slack workspace (haha which I actually created ages ago!)

To get started with Bactopia version 4, check out the installation guide and the tutorial.

AllTheBacteria​

Zamin Iqbal's Group, who brought us 661k bacterial assemblies, has now taken it a step further with AllTheBacteria. As someone once tasked with assembling "all the Staphylococcus aureus genomes" (although, it was only about 700 samples in 2010!), this is truly an impressive feat, and a valuable community resource! With the latest assemblies, the collection is now nearly 2,000,000 bacterial assemblies!

Similar to their previous methods, the latest version of AllTheBacteria uses Shovill for assembly. In addition, each assembly has basic metrics calculated, undergoes taxonomic abundance estimation, and has sketches made available. For more details about this project, please see:

• Preprint: AllTheBacteria - all bacterial genomes assembled, available and searchable
• GitHub: AllTheBacteria

Since Zamin revealed the latest updates on AllTheBacteria, I've been wondering: How could Bactopia users take advantage these assemblies? Especially, through available Bactopia Tools?

Why Bactopia Tools?​

The really nice thing about Bactopia Tools is they make it super easy to run 60 additional analyses on your genomes. It's really as simple as adding --wf <tool> to your Bactopia command, then Bactopia will then handle the rest for you, including container selection and audit trails.

Obviously, I'm a bit biased here, but utilizing Bactopia Tools in this situation would greatly streamline a lot of downstream analyses of AllTheBacteria assemblies. I think this would allow researchers to quickly get to the science behind these assemblies.

To give you an idea, there are currently 38 Bactopia Tools that use assemblies as inputs. In other words, each of these tools would be easy to run on the 2,000,000 AllTheBacteria assemblies.

bactopia atb-formatter​

Bactopia already allows assemblies as inputs, but I didn't want users to have to go through the full Bactopia pipeline to use the Bactopia Tools. Instead, I wanted to make a quick and easy way for users to go directly to using Bactopia Tools. To accomplish this, I created a new Bactopia command called atb-formatter (AllTheBacteria Formatter). With atb-formatter, the necessary Bactopia output directory structure will be created from a directory of _AllTheBacteria assemblies.

That's cool and all, but let's actually demonstrate the usage of atb-formatter on some Legionella pneumophila assemblies from AllTheBacteria.

Example Usage for Legionella pneumophila​

To demonstrate the usage of bactopia atb-formatter, I will use assemblies for Legionella pneumophila from AllTheBacteria and run legsta, a typing tool for L. pneumophila assemblies, written by Torsten Seeman, To be specific, I will run legsta from the available Bactopia Tool.

Getting Setup​

Before we get started, you'll need to have Bactopia installed. If you haven't done this yet, please see the installation instructions.

You will also want to make sure you are using at least version 3.0.1 of Bactopia, as this is the first release to have the atb-formatter command.

bactopia --version
bactopia 3.0.1

Download the Assemblies​

First I will download the L. pneumophila assemblies from AllTheBacteria, then extract them into a folder called legionella-assemblies. Easy enough!

mkdir atb-legionella
cd atb-legionella

# Download the assemblies
wget https://ftp.ebi.ac.uk/pub/databases/AllTheBacteria/Releases/0.1/assembly/legionella_pneumophila__01.asm.tar.xz
wget https://ftp.ebi.ac.uk/pub/databases/AllTheBacteria/Releases/0.1/assembly/legionella_pneumophila__02.asm.tar.xz

# Extract the assemblies
mkdir legionella-assemblies
tar -C legionella-assemblies -xJf legionella_pneumophila__01.asm.tar.xz
tar -C legionella-assemblies -xJf legionella_pneumophila__02.asm.tar.xz

At the time of writing this, there were 5,393 L. pneumophila assemblies available from AllTheBacteria. While its not Salmonella enterica with it's hundreds of thousands of assemblies, it's a great number to demonstrate the usage of bactopia atb-formatter.

Create the Bactopia Directory Structure​

With the assemblies extracted, now I need to create the required Bactopia directory to make use of Bactopia Tools. For this, I used bactopia atb-formatter, which creates a sample folder for each assembly that matches the BioSample accession.

# Create the Bactopia directory structure
bactopia atb-formatter --path legionella-assemblies --recursive

After running the above command, you should see something like the following:

2024-03-22 14:30:07:root:INFO - Setting up Bactopia directory structure (use --verbose to see more details)
2024-03-22 14:30:08:root:INFO - Bactopia directory structure created at bactopia
2024-03-22 14:30:08:root:INFO - Total assemblies processed: 5393

Use Bactopia to run Legsta​

Fancy! Now we have all the assemblies sym-linked into a Bactopia directory structure. It's time to let Bactopia Tools shine! To do this, I will run the legsta Bactopia Tool and demonstrate how seamless it is to type 5,393 assemblies.

With a simple addition of --wf legsta and pointing to the Bactopia directory, legsta will be executed on all 5,393 assemblies! It really is that simple!

# Run legsta
bactopia --wf legsta -profile singularity

After some time, the legsta tool will complete for all 5,393 assemblies, and you should be met with something like the following:

[5d/d04297] process > BACTOPIATOOLS:LEGSTA:LEGSTA_MODULE (SAMN29911258) [100%] 5393 of 5393 ✔
[71/c63bf7] process > BACTOPIATOOLS:LEGSTA:CSVTK_CONCAT (legsta)        [100%] 1 of 1 ✔
[16/833262] process > BACTOPIATOOLS:CUSTOM_DUMPSOFTWAREVERSIONS (1)     [100%] 1 of 1 ✔

    Bactopia Tools: `legsta Execution Summary
    ---------------------------
    Bactopia Version : 3.0.1
    Nextflow Version : 23.10.1
    Command Line     : nextflow run /home/rpetit3/bactopia/main.nf --wf legsta \
                                    --bactopia bactopia/ -profile singularity
    Resumed          : false
    Completed At     : 2024-03-22T15:09:54.959834620-06:00
    Duration         : 32m 51s
    Success          : true
    Exit Code        : 0
    Error Report     : -
    Launch Dir       : /home/rpetit3/test-legsta

This took about 30 minutes on my laptop, but it was incredibly simple to run legsta on all 5,393 L. pneumophila assemblies.

Results of Typing​

Here's the fun part, typing results for all 5,393 L. pneumophila assemblies! Another nice thing about the Bactopia Tools in that in most cases it will merge all the results at the end leaving you with just a single file to review.

To share the results of this analysis, I've uploaded the results to Google Drive and made them available from this link: Bactopia - Legsta Results from AllTheBacteria

From this Google Sheet, you can make a copy or export the results as a CSV file.

Conclusion​

In this post, I demonstrated how in just a few steps you can make use of Bactopia Tools to rapidly and seamlessly start analyzing the 2,000,000 bacterial assemblies from AllTheBacteria. If you are planning to do your own downstream analyses on these assemblies, I hope this post has convinced you that Bactopia can make this process much easier.

If you have any questions or ideas for additional Bactopia Tools, please feel free to reach out to me!

Also! This the first ever blog post for Bactopia!