magic-trace
casync
| magic-trace | casync | |
|---|---|---|
| 29 | 17 | |
| 4,472 | 1,478 | |
| 0.7% | 1.0% | |
| 6.8 | 2.4 | |
| about 1 month ago | 5 months ago | |
| OCaml | C | |
| MIT License | - |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
magic-trace
-
When Optimising Code, Measure
I really like magic-trace [0].
https://github.com/janestreet/magic-trace
Not that the exact tracing relies on Intel PT - support for AMD was added recently but uses perf so suffers from the same sampling/skew issues, but is still very useful.
-
Grafana Phlare, open source database for continuous profiling at scale
Would love to see this integrate with magic trace [1]. I'll need to look at the code for the flamegraph plugin, because handling nanosecond timestamps in flamegraphs seems to break most tools due to float precision.
(1) https://github.com/janestreet/magic-trace
-
How to break into Hudson River Trading?
As for inner workings, have you looked into magic trace? I want to play around with it but last I checked, it doesn't work on macs.
-
Brendan Intel.com
I really hope he can work with cloud vendors and Intel to make Processor Trace a more popular and easier to use capability.
It's unfortunate how https://github.com/janestreet/magic-trace and PMUs in general can't be used by lots of people using cloud VMs.
- GitHub - janestreet/magic-trace: magic-trace collects and displays high-resolution traces of what a process is doing
-
Hacker News top posts: Apr 23, 2022
Magic-trace – High-resolution traces of what a process is doing\ (133 comments)
- Magic-trace – High-resolution traces of what a process is doing
- Magic-trace - 高分辨率跟踪一个进程正在做什么 (Magic-trace – High-resolution traces of what a process is doing)
casync
-
We reduced conda’s index fetch bandwidth by 99%
For arbitrary state changes however, it's better to use something like casync. Note that there are a lot of tunables, implicit and explicit; for package indexing I would particularly think about "how is the index sorted" and "what is the desired chunk size".
-
Intro to Content Defined Chunking
If you just want something practical to play with, see casync. Even if it doesn't fit your workflow, or if you think you can do better, chances are you're best off building on top of it or adding patches to it, not starting from scratch.
-
Tool to clone file structure without the large files themselves?
You probably want casync.
-
A Nibble of Content-Defined Chunking - How de-duplicated, incremental file transfer works
Obligatory link to casync, which implements this better than most alternatives.
-
LibSQL – a fork of SQLite that is both Open Source, and Open Contributions
(personally, I think more people need to be aware of casync for the update storage/distribution problem. It isn't perfect for every use case, but it's good enough that you're probably better off wrapping/forking it rather than reimplementing it badly from scratch)
-
improving download infra
Does something like casync (https://github.com/systemd/casync or https://github.com/folbricht/desync) serve any purpose or provide any advantage to propagating rpm changes over rsync?
-
Are there any true alternatives to Seafile? (Nextcloud is not an alternative in this context)
Software that comes to mind for syncing lots of small files: git (and other source versioning tools), casync (https://github.com/systemd/casync) and a go implementation (https://github.com/folbricht/desync). Not really an answer and I can't think of a way to shoehorn that into your workflow, but maybe it leads you down a useful road.
- Casync – A Content-Addressable Data Synchronization Tool
-
Hacker News top posts: Apr 23, 2022
Casync – A Content-Addressable Data Synchronization Tool\ (15 comments)
What are some alternatives?
viztracer - VizTracer is a low-overhead logging/debugging/profiling tool that can trace and visualize your python code execution.
kopia - Cross-platform backup tool for Windows, macOS & Linux with fast, incremental backups, client-side end-to-end encryption, compression and data deduplication. CLI and GUI included.
perspective - A data visualization and analytics component, especially well-suited for large and/or streaming datasets.
tarsnap - Command-line client code for Tarsnap.
perfetto - Frontend for magic-trace; forks https://ui.perfetto.dev
desync - Alternative casync implementation
linux - Linux kernel source tree
zstd - Zstandard - Fast real-time compression algorithm
scalene - Scalene: a high-performance, high-precision CPU, GPU, and memory profiler for Python with AI-powered optimization proposals
BorgBackup - Deduplicating archiver with compression and authenticated encryption.
memray - Memray is a memory profiler for Python
rclone - "rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex Files