Compare commits
11 commits
Author | SHA1 | Date | |
---|---|---|---|
7f0abdbb93 | |||
381dcd1c1e | |||
4483828cf5 | |||
94e248b95e | |||
98f46c8081 | |||
d16f0957f9 | |||
cb36227bd4 | |||
cd620d648e | |||
44a33f2c72 | |||
b7325dfe07 | |||
c6105f8d61 |
3
.gitignore
vendored
3
.gitignore
vendored
|
@ -9,3 +9,6 @@ site
|
||||||
graph.png
|
graph.png
|
||||||
|
|
||||||
/preview
|
/preview
|
||||||
|
|
||||||
|
.jj
|
||||||
|
.direnv
|
||||||
|
|
18
Tuprules.tup
18
Tuprules.tup
|
@ -1,15 +1,17 @@
|
||||||
&root = .
|
&root = .
|
||||||
|
&tmpl_dir = ./templates
|
||||||
|
&gen_tmpl_dir = ./build/templates
|
||||||
|
|
||||||
HTML_TEMPLATE = main.html
|
HTML_TEMPLATE = main.html
|
||||||
|
|
||||||
!html = | &(root)/templates/footer.html \
|
!html = | &(tmpl_dir)/<tmpl_group> \
|
||||||
|> ^ html %f^ \
|
|> ^b html %f^ \
|
||||||
pandoc --from markdown --to html \
|
pandoc --from markdown --to html \
|
||||||
--template=&(root)/templates/$(HTML_TEMPLATE) \
|
--template=&(tmpl_dir)/$(HTML_TEMPLATE) \
|
||||||
|
--include-in-header=&(tmpl_dir)/header.html \
|
||||||
|
--include-before-body=&(tmpl_dir)/nav.html \
|
||||||
--css &(root)/css/style.css \
|
--css &(root)/css/style.css \
|
||||||
--include-in-header=&(root)/templates/header.html \
|
--include-after-body=&(gen_tmpl_dir)/footer.html \
|
||||||
--include-before-body=&(root)/templates/nav.html \
|
|
||||||
--include-after-body=&(root)/templates/footer.html \
|
|
||||||
%f | sed 's|%%webRoot%%|&(root)|g' > %o \
|
%f | sed 's|%%webRoot%%|&(root)|g' > %o \
|
||||||
|> %B.html
|
|> %B.html
|
||||||
|
|
||||||
|
@ -20,5 +22,5 @@ HTML_TEMPLATE = main.html
|
||||||
!compress_pdf = |> ps2pdf %f %o |>
|
!compress_pdf = |> ps2pdf %f %o |>
|
||||||
|
|
||||||
JPG_OPT = -quality 80 -strip -interlace Plane
|
JPG_OPT = -quality 80 -strip -interlace Plane
|
||||||
!blur_mini = |> convert %f -resize 400x400 -blur 0x8 $(JPG_OPT) %o |> blur_mini.jpg
|
!blur_mini = |> ^ %f -> blur mini^ convert %f -resize 400x400 -blur 0x8 $(JPG_OPT) %o |> blur_mini.jpg
|
||||||
!thumbnail = |> convert %f -resize 200x200 $(JPG_OPT) %o |>
|
!thumbnail = |> ^ %f -> thumbnail^ convert %f -resize 200x200 $(JPG_OPT) %o |>
|
||||||
|
|
24
about.md
24
about.md
|
@ -29,18 +29,15 @@ sail again.
|
||||||
In 2010 I moved to Oslo, Norway, to work as a [Qt](http://www.qt.io/) software
|
In 2010 I moved to Oslo, Norway, to work as a [Qt](http://www.qt.io/) software
|
||||||
engineer for [Nokia](http://www.nokia.com/) with the great people of Trolltech.
|
engineer for [Nokia](http://www.nokia.com/) with the great people of Trolltech.
|
||||||
|
|
||||||
But in the end, I decided Nokia's new direction with Windows Phone didn't suit
|
I learned a lot with such a bunch of talented people in different fields
|
||||||
me so I decided to fly back to France.
|
(graphics, web, filesystem, etc).
|
||||||
|
|
||||||
Damn I missed croissants.
|
|
||||||
|
|
||||||
### 2011 - Computer vision
|
### 2011 - Computer vision
|
||||||
|
|
||||||
In 2011 I had the chance to work on computer vision projects. I joined [LTU
|
In 2011 I had the chance to work on computer vision projects. I joined [LTU
|
||||||
technologies](http://www.ltutech.com/) as a research engineer in Paris, France.
|
technologies](http://www.ltutech.com/) as a research engineer in Paris, France.
|
||||||
I implemented exciting computer vision stuff (image matching, retrieval, etc),
|
I implemented exciting computer vision stuff (image matching, retrieval,
|
||||||
following the state of the art in the field. Unfortunately LTU shut down in
|
similarity, etc), following the state of the art in the field.
|
||||||
2015.
|
|
||||||
|
|
||||||
### 2015 - Camera software
|
### 2015 - Camera software
|
||||||
|
|
||||||
|
@ -52,15 +49,22 @@ working on the firmware level and I learned a ton about photography.
|
||||||
|
|
||||||
### 2017 - Maps
|
### 2017 - Maps
|
||||||
|
|
||||||
In 2017 I joined [Zenly](http://zen.ly). We made a social app with a strong
|
In 2017 I joined [Zenly](http://zen.ly) in Paris. We made a social app
|
||||||
focus on the map, putting the fun back in map apps.
|
with a strong focus on the map, putting the fun back in map apps. Although it
|
||||||
|
looked "simple", it involved a lot of technology, with a very custom tech stack.
|
||||||
|
|
||||||
### 2020 - Renewable energy
|
### 2020 - Renewable energy
|
||||||
|
|
||||||
In 2020 I joined [MoMA](https://www.momagroup.com/) to work on the
|
In 2020 I joined [MoMA](https://www.momagroup.com/) in Paris again, to work on the
|
||||||
[E6](https://www.e6-group.com/) project. I felt it was the right time to work on
|
[E6](https://www.e6-group.com/) project. I felt it was the right time to work on
|
||||||
current matters such as energy.
|
current matters such as energy.
|
||||||
|
|
||||||
|
### 2022 - Smart home
|
||||||
|
|
||||||
|
In 2022 I joined [Netatmo](https://www.netatmo.com/) in Boulogne-Billancourt to
|
||||||
|
work on smart homes. I work in the Vision team, making home cameras smarter. I
|
||||||
|
am thrilled to work on computer vision again.
|
||||||
|
|
||||||
<div style="text-align:center; padding-top:2em;">
|
<div style="text-align:center; padding-top:2em;">
|
||||||
<a href="https://octodon.social/@ffreling">
|
<a href="https://octodon.social/@ffreling">
|
||||||
<img src="images/mastodon.svg" alt="Mastodon" style="height:40px; padding-inline:5px">
|
<img src="images/mastodon.svg" alt="Mastodon" style="height:40px; padding-inline:5px">
|
||||||
|
|
110
articles/git_large_files.md
Normal file
110
articles/git_large_files.md
Normal file
|
@ -0,0 +1,110 @@
|
||||||
|
---
|
||||||
|
title: Git and large files
|
||||||
|
date: 2021-12-29
|
||||||
|
---
|
||||||
|
|
||||||
|
Git is a cornerstone of software development nowadays, it has become the
|
||||||
|
de-facto version control system.
|
||||||
|
|
||||||
|
Its interface is a bit complex to work with but a lot of tooling has been
|
||||||
|
developed over the years that lessen the pain to deal with it.
|
||||||
|
|
||||||
|
One shortcoming of git though (and version control system in general), is
|
||||||
|
dealing with huge binary files. These files are usually media assets that are
|
||||||
|
not meant to be diffable, but they do belong to the project nonetheless.
|
||||||
|
|
||||||
|
They are different ways to deal with these files:
|
||||||
|
|
||||||
|
### 1. treat them like regular text files
|
||||||
|
|
||||||
|
This is the easiest solution: do nothing special. It works perfectly and you
|
||||||
|
keep a clean history. However, as you modify your assets, the repository size
|
||||||
|
will grow and it will become slower to clone on your CI pipeline. It will also
|
||||||
|
put more charge on your git server.
|
||||||
|
|
||||||
|
### 2. keep them out of your repository
|
||||||
|
|
||||||
|
Out of the repository, out of trouble! If you keep your large assets in a
|
||||||
|
separate directory (Dropbox for instance), your repository will stay light. But
|
||||||
|
now you need to synchronize your external storage with your repository for your
|
||||||
|
project. Most of the time, only the latest version is kept around, making it
|
||||||
|
impossible to inspect an older revision with the appropriate assets.
|
||||||
|
|
||||||
|
### 3. store a pointer to external storage
|
||||||
|
|
||||||
|
As a compromise, you can store a pointer to external storage in your repository.
|
||||||
|
Everytime you checkout a specific revision, you will fetch the according data to
|
||||||
|
external storage and inject it into the project.
|
||||||
|
|
||||||
|
|
||||||
|
The solution 3. is the more convenient solution: we keep regular git workflow,
|
||||||
|
and put the burden of hosting large files out of git itself.
|
||||||
|
|
||||||
|
## Git Large File Storage
|
||||||
|
|
||||||
|
[Git Large File Storage](https://git-lfs.github.com/) (Git LFS) is the more
|
||||||
|
widespread implementation of this mecanism. It is developed by GitHub and is
|
||||||
|
available on all repositories on their platform. It works out of the box: you
|
||||||
|
set it once and you can forget about it.
|
||||||
|
|
||||||
|
However, there are some shortcomings with Git LFS.
|
||||||
|
|
||||||
|
### 1. your project is now longer self-contained in git
|
||||||
|
|
||||||
|
If you decide to use Git LFS, you will tie your project with the LFS storage
|
||||||
|
server. You won't be able to walk through your history without having a storage
|
||||||
|
server. GitHub LFS server implementation is currently closed-source and only a
|
||||||
|
"non production ready" reference server is available.
|
||||||
|
|
||||||
|
Major hosting platforms have implemented their own implementation and it is
|
||||||
|
possible to migrate your data among compatibles hosting platforms. But your
|
||||||
|
local copy of the repository will never hold all the data needed for your
|
||||||
|
project. In a way, the storage server becomes a centralized piece. You can fetch
|
||||||
|
all data locally to have it available but it won't be considered a source, it is
|
||||||
|
more like a cache.
|
||||||
|
|
||||||
|
### 2. you can't easily manage storage in LFS
|
||||||
|
|
||||||
|
If you commit a bunch of files, then push your changes, all the files will be
|
||||||
|
stored on the LFS server. If you want to remove them (eg. you uploaded unwanted
|
||||||
|
files), you can do it locally by doing a rebase, then call `git lfs prune`.
|
||||||
|
However, that will only clean up your local copies of files. What has been
|
||||||
|
pushed will stay on the server.
|
||||||
|
|
||||||
|
If you wish to remove files from the server, your options depend on the server
|
||||||
|
implementation:
|
||||||
|
- on GitHub, your only option to reclaim LFS quota and truly delete files from
|
||||||
|
LFS is to [delete and recreate your repository](https://docs.github.com/en/repositories/working-with-files/managing-large-files/removing-files-from-git-large-file-storage#git-lfs-objects-in-your-repository)
|
||||||
|
- on BitBucket Cloud, you can browse LFS files in the web UI and delete specific
|
||||||
|
files
|
||||||
|
|
||||||
|
## Git Annex
|
||||||
|
|
||||||
|
[git-annex](https://git-annex.branchable.com/) is a complete solution to deal
|
||||||
|
with external files in your git repository. It is also more complex than Git
|
||||||
|
LFS. As you can see in their
|
||||||
|
[walkthrough](https://git-annex.branchable.com/walkthrough/), you need to
|
||||||
|
explicitly set remotes for your files, and sync content between remotes.
|
||||||
|
|
||||||
|
Data is shared among local repositories in `.git/annex`, but it won't be
|
||||||
|
available in common source forges such as GitHub. To make this data available to
|
||||||
|
all people in the project, you can use [special
|
||||||
|
remotes](https://git-annex.branchable.com/special_remotes/) which are used as
|
||||||
|
data storage stores, akin to Git lFS (which can be used as a special remote).
|
||||||
|
|
||||||
|
Contrary to Git LFS, you can see what content is currently
|
||||||
|
[unused](https://git-annex.branchable.com/walkthrough/unused_data/), [delete
|
||||||
|
unwanted files](https://git-annex.branchable.com/tips/deleting_unwanted_files/).
|
||||||
|
It is a more complex solution but it is more flexible.
|
||||||
|
|
||||||
|
## What I recommend
|
||||||
|
|
||||||
|
I think git-annex gives the user more control over its data: it can be fully
|
||||||
|
decentralized and offers tools to manage its content.
|
||||||
|
|
||||||
|
Git LFS is simpler and more widely used, but once you hit one of its limitation,
|
||||||
|
it can be costly to break free.
|
||||||
|
|
||||||
|
## Links
|
||||||
|
|
||||||
|
- [Large files with Git: LFS and git-annex](https://lwn.net/Articles/774125/)
|
25
flake.lock
Normal file
25
flake.lock
Normal file
|
@ -0,0 +1,25 @@
|
||||||
|
{
|
||||||
|
"nodes": {
|
||||||
|
"nixpkgs": {
|
||||||
|
"locked": {
|
||||||
|
"lastModified": 1731890469,
|
||||||
|
"narHash": "sha256-D1FNZ70NmQEwNxpSSdTXCSklBH1z2isPR84J6DQrJGs=",
|
||||||
|
"owner": "NixOS",
|
||||||
|
"repo": "nixpkgs",
|
||||||
|
"rev": "5083ec887760adfe12af64830a66807423a859a7",
|
||||||
|
"type": "github"
|
||||||
|
},
|
||||||
|
"original": {
|
||||||
|
"id": "nixpkgs",
|
||||||
|
"type": "indirect"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"root": {
|
||||||
|
"inputs": {
|
||||||
|
"nixpkgs": "nixpkgs"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"root": "root",
|
||||||
|
"version": 7
|
||||||
|
}
|
19
flake.nix
Normal file
19
flake.nix
Normal file
|
@ -0,0 +1,19 @@
|
||||||
|
{
|
||||||
|
description = "Website";
|
||||||
|
|
||||||
|
outputs = { self, nixpkgs }:
|
||||||
|
let pkgs = nixpkgs.legacyPackages.x86_64-linux;
|
||||||
|
in {
|
||||||
|
devShell.x86_64-linux = with pkgs;
|
||||||
|
mkShell {
|
||||||
|
nativeBuildInputs = [
|
||||||
|
imagemagick
|
||||||
|
just
|
||||||
|
libavif
|
||||||
|
libjxl
|
||||||
|
pandoc
|
||||||
|
tup
|
||||||
|
];
|
||||||
|
};
|
||||||
|
};
|
||||||
|
}
|
2
justfile
2
justfile
|
@ -19,7 +19,7 @@ upload-resume:
|
||||||
deploy: build
|
deploy: build
|
||||||
rsync --checksum --copy-links -ave 'ssh' \
|
rsync --checksum --copy-links -ave 'ssh' \
|
||||||
--exclude-from=rsync_excludes.txt \
|
--exclude-from=rsync_excludes.txt \
|
||||||
build/* fabs@ffreling.com:ffreling.com/
|
build/* fabs@ffreling.com:/var/www/ffreling.com/
|
||||||
|
|
||||||
preview: build
|
preview: build
|
||||||
python3 -m webbrowser -t "file://{{root}}/build/index.html"
|
python3 -m webbrowser -t "file://{{root}}/build/index.html"
|
||||||
|
|
12
shell.nix
Normal file
12
shell.nix
Normal file
|
@ -0,0 +1,12 @@
|
||||||
|
{ pkgs ? import <nixpkgs> {} }:
|
||||||
|
|
||||||
|
pkgs.mkShell {
|
||||||
|
buildInputs = with pkgs; [
|
||||||
|
imagemagick
|
||||||
|
just
|
||||||
|
libavif
|
||||||
|
nushell
|
||||||
|
pandoc
|
||||||
|
tup
|
||||||
|
];
|
||||||
|
}
|
|
@ -1 +1 @@
|
||||||
: |> ./generate_footer.sh > %o |> footer.html
|
: |> ./generate_footer.sh > %o |> footer.html ./<tmpl_group>
|
||||||
|
|
|
@ -5,3 +5,4 @@
|
||||||
<a href="%webRoot%/articles/index.html">Articles</a>
|
<a href="%webRoot%/articles/index.html">Articles</a>
|
||||||
<a href="%webRoot%/about.html">About</a>
|
<a href="%webRoot%/about.html">About</a>
|
||||||
</div>
|
</div>
|
||||||
|
<hr>
|
||||||
|
|
Loading…
Reference in a new issue