Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
toolbox/github/data
toolbox/diver/data
toolbox/gitlab/data
toolbox/utils/_templates/new-GitHub-repository-contributors.fr.template.txt.result
83 changes: 42 additions & 41 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,142 +7,143 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## Unreleased

- Make CHANGELOG more compliant (#103)
- Generate template-based text using variables ([#84](https://github.com/Orange-OpenSource/floss-toolbox/issues/84))
- Make CHANGELOG more compliant ([#103](https://github.com/Orange-OpenSource/floss-toolbox/issues/103))

## [2.13.0] - 2023-07-19

### Added

- Diver - Compute metrics with in parameter URL to clone repo (#98)
- Diver - Compute metrics with in parameter URL to clone repo ([#98](https://github.com/Orange-OpenSource/floss-toolbox/issues/98))

### Fixed

- Project - Broken links in README (#96)
- Project - Broken links in README ([#96](https://github.com/Orange-OpenSource/floss-toolbox/issues/96))

## [2.12.0] - 2023-07-18

### Added

- Diver - Lines of codes and useful metrics (#28)
- Diver - Lines of codes and useful metrics ([#28](https://github.com/Orange-OpenSource/floss-toolbox/issues/28))

### Changed

- Project - Add DCO (#87)
- Project - Add security policy file (#90)
- Project - Split README files (#85)
- Licenses Inventory - Move HTML test files to archives of release (#86)
- GitHub - Add in dry-run Gemfiles (#93)
- Project - Add DCO ([#87](https://github.com/Orange-OpenSource/floss-toolbox/issues/87))
- Project - Add security policy file ([#90](https://github.com/Orange-OpenSource/floss-toolbox/issues/90))
- Project - Split README files ([#85](https://github.com/Orange-OpenSource/floss-toolbox/issues/85))
- Licenses Inventory - Move HTML test files to archives of release ([#86](https://github.com/Orange-OpenSource/floss-toolbox/issues/86))
- GitHub - Add in dry-run Gemfiles ([#93](https://github.com/Orange-OpenSource/floss-toolbox/issues/93))

## [2.11.0] - 2023-06-28

### Added

- GitHub - Set teams permissions to read (#82)
- GitHub - Set teams permissions to read ([#82](https://github.com/Orange-OpenSource/floss-toolbox/issues/82))

## [2.10.1] - 2023-05-31

### Fixed

- Diver - Path variables not protected (#80)
- Diver - Path variables not protected ([#80](https://github.com/Orange-OpenSource/floss-toolbox/issues/80))

## [2.10.0] - 2023-05-30

### Added

- Licenses Inventory - New release (#77)
- Licenses Inventory - New release ([#77](https://github.com/Orange-OpenSource/floss-toolbox/issues/77))

## [2.9.0] - 2023-03-31

### Added

- Licenses Inventory - New release (#64)
- Licenses Inventory - New release ([#64](https://github.com/Orange-OpenSource/floss-toolbox/issues/64))

## [2.8.0] - 2023-03-10

### Added

- Project - Split dry run (#68)
- Project - Split dry run ([#68](https://github.com/Orange-OpenSource/floss-toolbox/issues/68))

### Changed

- Project - Update copyright (#70)
- Project - Improve README (#69)
- Project - Update copyright ([#70](https://github.com/Orange-OpenSource/floss-toolbox/issues/70))
- Project - Improve README ([#69](https://github.com/Orange-OpenSource/floss-toolbox/issues/69))

## [2.7.1]

### Changed

- Add missing files (#63)
- Add missing files ([#63](https://github.com/Orange-OpenSource/floss-toolbox/issues/63))

## [2.7.0] - 2023-01-18

### Added

- Package manager - Extract from files downloaded dependencies (#2)
- Package manager - Extract from files downloaded dependencies ([#2](https://github.com/Orange-OpenSource/floss-toolbox/issues/2))

## [2.6.0] - 2022-05-05

### Added

- Look for leaks and vulnerabilities with exclusion of projects (#57)
- Look for leaks and vulnerabilities with exclusion of projects ([#57](https://github.com/Orange-OpenSource/floss-toolbox/issues/57))

## [2.5.0] - 2022-03-09

### Added

- GitLab Auto Backup (#32)
- Look for leaks (GitLab) (#49)
- GitLab Auto Backup ([#32](https://github.com/Orange-OpenSource/floss-toolbox/issues/32))
- Look for leaks (GitLab) ([#49](https://github.com/Orange-OpenSource/floss-toolbox/issues/49))

### Fixed

- Failure of git log if no commits (#52)
- Failure of git log if no commits ([#52](https://github.com/Orange-OpenSource/floss-toolbox/issues/52))

## [2.4.0] - 2022-03-08

### Added

- Look for leaks (GitHub) (#44)
- Dry run (#29)
- Look for leaks (GitHub) ([#44](https://github.com/Orange-OpenSource/floss-toolbox/issues/44))
- Dry run ([#29](https://github.com/Orange-OpenSource/floss-toolbox/issues/29))

### Changed

- Check of vulnerabilities (#37)
- Fix typo in doc and files (#40)
- Check of vulnerabilities ([#37](https://github.com/Orange-OpenSource/floss-toolbox/issues/37))
- Fix typo in doc and files ([#40](https://github.com/Orange-OpenSource/floss-toolbox/issues/40))

## [2.3.0] - 2022-02-25

### Added

- Find repositories with vulnerabilities (Dependabot) (#20)
- Find repositories with vulnerabilities (Dependabot) ([#20](https://github.com/Orange-OpenSource/floss-toolbox/issues/20))

## [2.2.0] 2022-02-24

### Added

- Backup of repositories (#19)
- Extract email addresses (#27)
- Backup of repositories ([#19](https://github.com/Orange-OpenSource/floss-toolbox/issues/19))
- Extract email addresses ([#27](https://github.com/Orange-OpenSource/floss-toolbox/issues/27))

## [2.1.0] - 2021-10-06

### Added

- List all contributors of a Git repository using Git history (#13)
- List all contributors of a Git repository using Git history ([#13](https://github.com/Orange-OpenSource/floss-toolbox/issues/13))

## [2.0.0] - 2021-06-05

## Added

- Get all members of GitHub organization (#4)
- Get members who don't have 2FA enabled (#4)
- Get members of organization with "company" field undefined (#4)
- Get projects which don't have any assigned GitHub team (#4)
- Get users which have undefined or hidden email (#4)
- Get users which may have not suitable fullname (#4)
- Get repositories with undefined licenses (#4)
- Get repositories which seems to be unconform (i.e. missing files) (#4)
- Get repositories which seems to be empty or have not enough files (#4)
- Define permissions (push / write) for all contributors of all projects (except teams and organization owners) (#4)
- Define permissions (push / write) for all teams of all projects (#4)
- Get all members of GitHub organization ([#4](https://github.com/Orange-OpenSource/floss-toolbox/issues/4))
- Get members who don't have 2FA enabled ([#4](https://github.com/Orange-OpenSource/floss-toolbox/issues/4))
- Get members of organization with "company" field undefined ([#4](https://github.com/Orange-OpenSource/floss-toolbox/issues/4))
- Get projects which don't have any assigned GitHub team ([#4](https://github.com/Orange-OpenSource/floss-toolbox/issues/4))
- Get users which have undefined or hidden email ([#4](https://github.com/Orange-OpenSource/floss-toolbox/issues/4))
- Get users which may have not suitable fullname ([#4](https://github.com/Orange-OpenSource/floss-toolbox/issues/4))
- Get repositories with undefined licenses ([#4](https://github.com/Orange-OpenSource/floss-toolbox/issues/4))
- Get repositories which seems to be unconform (i.e. missing files) ([#4](https://github.com/Orange-OpenSource/floss-toolbox/issues/4))
- Get repositories which seems to be empty or have not enough files ([#4](https://github.com/Orange-OpenSource/floss-toolbox/issues/4))
- Define permissions (push / write) for all contributors of all projects (except teams and organization owners) ([#4](https://github.com/Orange-OpenSource/floss-toolbox/issues/4))
- Define permissions (push / write) for all teams of all projects ([#4](https://github.com/Orange-OpenSource/floss-toolbox/issues/4))

## [1.0.0]

Expand Down
3 changes: 3 additions & 0 deletions CODEOWNERS
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,14 @@ CODE_OF_CONFLICT.md @pylapp
CONTRIBUTORS.txt @pylapp
LICENSE.txt @pylapp
README.md @pylapp
DCO.txt @pylapp
SECURITY.md @pylapp

# Toolbox

/toolbox/LicensesInventory/ @pylapp @laurentora
/toolbox/diver/ @pylapp
/toolbox/github/ @pylapp
/toolbox/gitlab/ @pylapp
/toolbox/utils/ @pylapp
/toolbox/dry-run.sh @pylapp
21 changes: 13 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,29 +7,34 @@
[![Shell](https://img.shields.io/badge/-Shell-89e051?style=for-the-badge)](https://github.com/Orange-OpenSource/floss-toolbox/search?l=shell)
[![Python](https://img.shields.io/badge/-Python-3572A5?style=for-the-badge)](https://github.com/Orange-OpenSource/floss-toolbox/search?l=python)
[![Ruby](https://img.shields.io/badge/-Ruby-701516?style=for-the-badge)](https://github.com/Orange-OpenSource/floss-toolbox/search?l=ruby)
[![PHP](https://img.shields.io/badge/-PHP-4F5B93?style=for-the-badge)](https://github.com/Orange-OpenSource/floss-toolbox/search?l=php)

# FLOSS Toolbox

Toolbox to help developers and open source referents to have cleaner projects in _GitHub_ organizations, and more.

Toolbox is mainly written in _Shell_ because this language is very efficient for files processing and provides a strong and rich standard API with cool primitives. It helps also to call system primitives easily. Contains also _Ruby_ scripts. _Ruby_ are shiny gems, I love them. _Python_ is also used.
Toolbox is mainly written in _Shell_ because this language is very efficient for files processing and provides a strong and rich standard API with cool primitives and nice performances due to system calls. It helps also to call system primitives easily.
Contains also _Ruby_ scripts. _Ruby_ are shiny gems, I love them.
_Python_ is also used.
And a bit of _PHP_ because it is nice to use several languages we are not used to (stop the routine!).
For these needs scripting is enough.

# Environment

You must have a _BASH_ ready environment and also _Ruby_.
Environment:
You should have mainly the following environments bellow, but have a look on each folder README:
- _Bash_ version **3.2.5**
- _Ruby_ version **2.7.1**
- _Python_ version **3.7**

# Project tree

There are four folders containing scripts and programs to make your life a bit easier:
There are 5 folders containing scripts and programs to make your life a bit easier:

1. _toolbox/diver_ contains scripts to scrap data in Git logs and histories, look for sensitive data in sources, etc.
2. _toolbox/github_ contains scripts and programs to make requests to GitHub API so as to automate some actions
3. _toolbox/gitlab_ contains scripts and programs to make requests to GitLab API so as to automate some actions
4. _toobox/LicensesInventory_ contains program to get licenses of third party components thanks to dependency manager files
1. _toolbox/diver_ contains scripts to scrap data in Git logs and histories, look for sensitive data in sources, etc ;
2. _toolbox/github_ contains scripts and programs to make requests to GitHub API so as to automate some actions ;
3. _toolbox/gitlab_ contains scripts and programs to make requests to GitLab API so as to automate some actions ;
4. _toolbox/LicensesInventory_ contains program to get licenses of third party components thanks to dependency manager files ;
5. _toolbox/utils_ contains scripts to generate texts and stuff like that.

Feel free to read each README available in all of the subdirectories listed above.

Expand Down
14 changes: 13 additions & 1 deletion toolbox/dry-run.sh
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@

# Since...............: 08/03/2021
# Description.........: Make a dry-run of the project to check if everything is ready to use
# Version.............: 3.0.0
# Version.............: 3.1.0

set -eu

Expand Down Expand Up @@ -84,6 +84,18 @@ echo "----------------------------------"

RunDryRunInFolder "gitlab"

# Utils features
# --------------

echo -e "\n"
read -p "✋ Press any key to dry run utils features"

echo -e "\n---------------------------------"
echo "Assertions for the Utils features"
echo "---------------------------------"

RunDryRunInFolder "utils"

# Licenses Inventory tool
# -----------------------

Expand Down
35 changes: 35 additions & 0 deletions toolbox/utils/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# Utils

# Generate text from template

## Prerequisites

- PHP

## Install prerequisites

```shell
brew install php
```

## Description

Sometimes as open source reference or software forges administrator, we need so send emails to people.
These emails can be big with lot of details and hyperlinks to resources, and writing them is time expansive.
Because these emails are almost the same (except with some details), we can generate them using a template and variables.

```shell
php text-generator.php "_templates/new-GitHub-repository-contributors.fr.template.txt" "_templates/values.ini"
```

Here we give to the `text-generator.php` PHP script a template to process (first argument) and also an .ini file containing values (second argument). The script will check if all variables are filled, then will replace each entry in the text by the values.

For example, if we have such .ini file:

```text
[some_section_which_is_ignored]
; Some comment
VARIABLE_NAME = "Foo-Bar"
```

the script will replace all occurences of **%VARIABLE_NAME%** by "Foo-Bar" in the text. Then a file with the new version will be created with quite the same name but _.result_ at the end.
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
Bonjour,

Les utilisateurs GitHub ci-dessous seront ajoutés avec les droits "%FORGE_ADDED_USERS_PERMISSION%" au dépôt GitHub "%FORGE_REPO_NAME%" [1] :
%FORGE_ADDED_HANDLES_LIST%.

Normalement ces droits seront suffisants mais nous vous invitons à nous contacter si ce n’est pas le cas en fournissant l'argumentaire adéquat.

Les utilisateurs ont été ajoutés à une team GitHub ("%FORGE_REPO_TEAM_NAME%"), mais c’est pour notre tambouille interne côté administrateurs.
Cette team, pour les mêmes raisons, a été ajoutée avec les droits "%FORGE_REPO_TEAM_PERMISSION%" au dépôt, il faudra laisser les choses en l'état.
Les utilisateurs ont été ajoutés à notre liste de diffusion des usagers des forges logicielles publiques (fréquence faible de méls envoyés) (%RESSOURCE_MAILING_LIST_FORGE_USERS%).

Les utilisateurs devront recevoir sous peu des invitations méls pour plusieurs sujets :
- Rejoindre l'organisation GitHub du Groupe (%FORGE_ORG_NAME%) ;
- Rejoindre le dépôt (%FORGE_REPO_NAME%).

Les comptes GitHub doivent respecter les règles ci-dessous, ceci ne prend que quelques minutes et c'est indispensable de le faire.
Les administrateurs de l'organisation n’ayant pas la main sur les comptes, leurs propriétaires doivent s'en charger, condition impérative pour utiliser les dépôts.
1. Avoir son prénom et son nom clairement affichés ;
2. Afficher dans le champ "company" votre entreprise (e.g. "Orange") ;
3. Afficher son appartenance à l’organisation GitHub (cf cette documentation [2]) ;
4. Ajouter son adresse email professionnelle parmi les adresses emails de votre compte GitHub ; attention, votre compte étant personnel, ne mettez pas votre adresse email professionnelle en tant qu’adresse "primary".

À noter que seuls les internes (CDI, apprentis, thésards) sont concernés par les critères susmentionnés, et non les prestataires et stagiaires qui ne peuvent rejoindre l’organisation (mais peuvent être ajoutés aux dépôts).

Concernant vos contributions à venir, nous vous demanderons d’être vigilants à votre configuration Git (prénom et nom correctement renseignés dans le "user.name", adresse mail pro correctement renseignée dans le "user.email").
Nous vous demanderons aussi d’appliquer le DCO (option signed-off de votre client Git).
Pour les externes, si vous avez une adresse email Orange, il faut l’utiliser. Sinon ce sera votre adresse email employeur.

Bien évidemment, vous devriez apporter le plus grand soin à vos commits, et être vigilants quant à la correction des vulnérabilités et fuites de secrets.

Par défaut les personnes auront les droits "%FORGE_ADDED_USERS_PERMISSION%", nous n’autorisons le droit "admin" qu’au cas par cas compte-tenu des possibilités de cette permission.
Nous nous réservons le droit de retirer ces droits à tout moment sans préavis si les personnes ne respectent pas les règles.

Pour rappel, les CGU de GitHub [3] n’autorisent pas d’avoir plusieurs comptes GitHub gratuits, et Orange ne fournit aucun compte GitHub à ses employés, vous devrez donc utiliser votre compte GitHub personnel pour contribuer si vous en avez un.
Pour plus d’information, nous vous invitons à consulter ce document [4] qui reprend les points à traiter concernant vos comptes GitHub.

Ne pas oublier non plus de signer ses commits avec le signoff [5] (les signatures cryptographiques via GPG ne sont pas exigées mais rien n'empêche de le faire).
Il faudra aussi être vigilant quant aux alertes Dependabot [6] et aux fuites de secrets détectables via gitleaks [7].

Pensez également à avoir les en-têtes des fichiers sources à jour [8] et à maintenir au besoin le fichier listant les dépendances tierces [9].
L'ajout d'apps tierces ou de GitHub Actions aux dépôts est soumis à validation auprès des administrateurs de l'organisation, prévenez-nous quand vous faîtes de telles demandes car nous n'avons que des informations parcellaires.
Pour ce sujet ou toute question, vous pouvez nous contacter via %RESSOURCE_MAILING_LIST_ADMINS%.

Pour aller plus loin, nous avons :
- un espace %TOOL_CHAT_NAME% pour discuter [10] ;
- un espace %TOOL_DOCS_NAME% dédié à l'open source [11] ;
- un espace %TOOL_DOCS_NAME% dédié à la propriété intellectuelle [12] ;
- un site web interne [11] ;
- un site web externe [12] ;
- un site vitrine pour GitHub [13].

Restant à disposition,

[1] %FORGE_REPO_URL%
[2] https://docs.github.com/fr/account-and-profile/setting-up-and-managing-your-personal-account-on-github/managing-your-membership-in-organizations/publicizing-or-hiding-organization-membership
[3] https://docs.github.com/en/site-policy/github-terms/github-terms-of-service#3-account-requirements
[4] %RESSOURCE_DOC_GITHUB-ACCOUNT-SETTINGS%
[5] %RESSOURCE_DOC_COMMITS-SIGNING%
[6] %RESSOURCE_DOC_DEPENDABOT%
[7] %RESSOURCE_DOC_GITLEAKS%
[8] %RESSOURCE_DOC_ABOUT_HEADERS%
[9] %RESSOURCE_DOC_ABOUT_THIRD-PARTY%
[10] %TOOL_CHAT_LINK%
[11] %TOOL_DOCS_OPENSOURCE_LINK%
[12] %TOOL_DOCS_IPJD_LINK%
[11] %RESSOURCES_WEB_INTERNAL-SITE%
[12] %RESSOURCES_WEB_PUBLIC-SITE%
[13] %RESSOURCES_WEB_GITHUB-SITE%
Loading