So, I've been using Thanos to receive and store my prometheus metrics long term in a self hosted S3 bucket. Thanos also acts as a datasource for my dashboards in Grafana, and provides a Ruler, which evaluates alerting rulers and forwards them to my alertmanager. It's ok. It's certainly got it's downsides, which I can go into later, but I've thinking... what about Mimir?

How do you all feel about Grafana's Mimir (source on GitHub)? It's AGPL and seems to literally be a replacement of Thanos, which is Apache 2.0.

Thanos description from their website:

Open source, highly available Prometheus setup with long term storage capabilities.

Mimir description from their website:

...open source software project that provides horizontally scalable, highly available, multi-tenant, long-term storage for Prometheus and OpenTelemetry metrics.

Both with work with alloy and prometheus alike. Both require you to configure initially confusing hashrings and replication parameters. Both have a bunch of large companies adopting them, so... now I feel conflicted. Should I try mimir? Poll in reply.

Hello, hachyderm! we've been working hard on building up our ansible runbooks and improving hachyderm's overall resilience. Recently, we've been focusing on is database resilience.

We're getting close to retiring our original database server (finally!) and preparing to move to a fully ansible-managed set of databases servers, primary and replica on new hardware. We'll send another announcement when we do the cut over. The team has done excellent work to make this highly automated, quick, and painless!

Done:

author ansible roles for managing postgresql, pgbackrest (backups), pgbouncer, and primary/replica failover
decide to continue with pgbouncer and *not* use pgcat
rotate database passwords
order new replica database hardware
order new future primary database hardware

To do soon:

rebuild replica database with ansible scripts
prepare primary database with ansible scripts
start replicating to new database replica
cut over to new database server

We're also planning on open-sourcing our ansible roles in the coming weeks - just a little housekeeping & tidying up before we do!

And that concludes my job in infosec.
Tomorrow, I will be in IT working on Linux.
#infosec #cybersecurity #SRE #linux #foss

So hand-wavy ; you still need #OnCall for critical services with defined SLAs :

“Breaking Up With On-Call”, Alexey Karandashev (https://reflector.dev/articles/breaking-up-with-on-call/).

Via HN: https://news.ycombinator.com/item?id=43378671

howdy, #hachyderm!

over the last week or so, we've been preparing to move hachy's #DNS zones from #AWS route 53 to bunny DNS.

since this could be a pretty scary thing -- going from one geo-DNS provider to another -- we want to make sure *before* we move that records are resolving in a reasonable way across the globe.

to help us to do this, we've started a small, lightweight tool that we can deploy to a provider like bunny's magic containers to quickly get DNS resolution info from multiple geographic regions quickly. we then write this data to a backend S3 bucket, at which point we can use a tool like #duckdb to analyze the results and find records we need to tweak to improve performance. all *before* we make the change.

then, after we've flipped the switch and while DNS is propagating -- -- we can watch in real-time as different servers begin flipping over to the new provider.

we named the tool hachyboop and it's available publicly --> https://github.com/hachyderm/hachyboop

please keep in mind that it's early in the booper's life, and there's a lot we can do, including cleaning up my hacky code.

attached is an example of a quick run across 17 regions for a few minutes. the data is spread across multiple files but duckdb makes it quite easy for us to query everything like it's one table.

hello, #SRE friends. looking for good resources for SRE-related content, specifically for #observability please share your fave resources, TY

There'd be a lot less whining from devs about 'needing' root in prod if K & R had named that account janitor

@sre @devops #sre #devops

My younger daughter wants to interview for a Lead Staff SRE position at a new company (but for an old boss). She'd like some ideas of what she might expect.

This is work she's currently doing at a large company, but due to the way her career has unfolded, she's never actually done a coding interview before.

There's 2 rounds, qualifying, and final.
Qualifying is leetcode stuff, data structures etc.
Final one is 4 parts, each one hour:

• behavioral
• Python/code review
• system design
• Linux

She's worried about coding for someone for the first time. And what algorithms/data structures they'll want (she doesn't have a formal CS background). She also isn't sure (nor am I) what the Linux segment would be likely to cover.

I know she has the skills and can talk through the problems well. She's survived lots of layoffs because she's the one who makes everything work and builds strong teams with focused goals. But now she's the last person left in her group and training folks in India, so the clock is ticking.

I can DM a link to the job description if that helps. Would rather not publish details publicly.

Thanks.

howdy, folks - it's been a bit since our last #hachyderm infra check in.

stuff in motion:

- ditching #terraform cloud & tf for #opentofu and #atlantis. we are just about to import our dev environment and put it through its paces.
- bringing #postgresql under ansible management. the team has been doing awesome work, and we've started to spin up dev nodes using the new playbooks. soon: production!
- moving #DNS zones away from AWS route 53. we chose bunny DNS as our provider and have been doing basic tests in dev. we'll likely prep our records for production this week with a plan for a cutover in one of the coming weekends.

and if you filled out our volunteer form and haven't heard from me in a bit - you're still on the list. we'll onboard a new batch of folks in the next couple of weeks.

@hachyderm

My employer is an EFF-approved independent ISP, and we're hiring a Senior Systems Engineer https://grnh.se/a6b53dbd1us
Key technologies involved: Linux (RedHat), MySQL/MariaDB, VMware,Proxmox, Ansible, Perl, Python and Shell.

Salary starts at $145k. Can be fully remote (within US), but preference if able to work in our northern california office one day a week.

See link for details (and better accuracy).
#syseng #sysadmin #sre #devops #hiring

Bueno, venga, va, nunca he hecho #presentación a pesar de que me he mudado más veces de las que puedo contar.

Soy #DevOps, #SRE o #PlatformEngineer, depende de cómo vaya el día. Mi trabajo no me define, pero me flipan los ordenadores desde lechón, así que un poco si.

Me gusta la broma y la chanza, pero si te ofendo con algún chiste, avísame para cambiarlo.

No me interesaba la política hasta que LOS PUTOS FASCISTAS me obligaron.

Para lo demás, la bio, o pregunta.

Hahaha
https://github.com/learnk8s/xlskubectl