Skip to content

Commit 0ad2d41

Browse files
authored
Version 2.8.0 (#74)
* chore: project - update README Add more badges Signed-off-by: Pierre-Yves Lapersonne <[email protected]> * #68] [Feature] Split dry-run scripts into modules #71 Signed-off-by: Pierre-Yves Lapersonne <[email protected]> * [#70] [Change] Project - Update copyright #72 Signed-off-by: Pierre-Yves Lapersonne <[email protected]> * [#69] [Doc] Improve README #73 Signed-off-by: Pierre-Yves Lapersonne <[email protected]> --------- Signed-off-by: Pierre-Yves Lapersonne <[email protected]>
1 parent 0ceab1b commit 0ad2d41

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

56 files changed

+856
-321
lines changed

CHANGELOG.md

Lines changed: 19 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,33 @@
11
# CHANGELOG
22

3+
## Version 2.8.0
4+
5+
### Features
6+
7+
- [#68](https://github.com/Orange-OpenSource/floss-toolbox/issues/68) Project - Split dry run
8+
9+
### Changes
10+
11+
- [#70](https://github.com/Orange-OpenSource/floss-toolbox/issues/70) Project - Update copyright
12+
- [#69](https://github.com/Orange-OpenSource/floss-toolbox/issues/69) Project - Improve README
13+
14+
## Version 2.7.1
15+
16+
### Changes
17+
18+
- [#63](https://github.com/Orange-OpenSource/floss-toolbox/issues/63) Add missing files
19+
320
## Version 2.7.0
421

522
### Features
623

7-
- [#2](https://github.com/Orange-OpenSource/floss-toolbox/issues/2) Package manager - Extract from files downloaded dependencies
24+
- [#2](https://github.com/Orange-OpenSource/floss-toolbox/issues/2) Package manager - Extract from files downloaded dependencies
825

926
## Version 2.6.0
1027

1128
### Features
1229

13-
- [#57](https://github.com/Orange-OpenSource/floss-toolbox/issues/57) Look for leaks and vulnerabilities with exclusion of projects
30+
- [#57](https://github.com/Orange-OpenSource/floss-toolbox/issues/57) Look for leaks and vulnerabilities with exclusion of projects
1431

1532
## Version 2.5.0
1633

README.md

Lines changed: 64 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,10 @@
33
[![Versions](https://img.shields.io/github/v/release/Orange-OpenSource/floss-toolbox?label=Last%20version&style=for-the-badge)](https://github.com/Orange-OpenSource/floss-toolbox/releases)
44
[![Still maintained](https://img.shields.io/maintenance/yes/2023?style=for-the-badge)](https://github.com/Orange-OpenSource/floss-toolbox/issues?q=is%3Aissue+is%3Aclosed)
55

6+
[![Shell](https://img.shields.io/badge/-Shell-89e051?style=for-the-badge)](https://github.com/Orange-OpenSource/floss-toolbox/search?l=shell)
7+
[![Python](https://img.shields.io/badge/-Python-3572A5?style=for-the-badge)](https://github.com/Orange-OpenSource/floss-toolbox/search?l=python)
8+
[![Ruby](https://img.shields.io/badge/-Ruby-701516?style=for-the-badge)](https://github.com/Orange-OpenSource/floss-toolbox/search?l=ruby)
9+
610
# FLOSS Toolbox
711

812
Toolbox to help developers and open source referents to have cleaner projects in _GitHub_ organizations, and more.
@@ -34,6 +38,64 @@ To be sure you have a ready-to-run project, you can run the dry-run command:
3438
bash dry-run.sh
3539
```
3640

41+
Table of Contents
42+
=================
43+
* [The diver of source code and commits](#the-diver-of-source-code-and-commits)
44+
* [Input file](#file-of-words)
45+
* [Features](#features)
46+
* [Find contributors in files](#find-contributors-in-files)
47+
* [Find contributors in Git log](#find-contributors-in-git-log)
48+
* [Find credentials in sources](#find-credentials-in-sources)
49+
* [Find missing signed-off fields](#find-credentials-in-sources)
50+
* [Find credits](#find-credits)
51+
* [List conributors from Git history](#list-contributors-from-git-history)
52+
* [Extract email adress from Git history](#extract-email-address-from-git-history)
53+
* [Play with GitHub web API](#play-with-github-web-api)
54+
* [Prerequisites](#prerequisites)
55+
* [Prepare project](#prepare-project)
56+
* [Third party elements](#third-party-elements)
57+
* [Features](#features-1)
58+
* [Display usages](#display-usage)
59+
* [Get all members of organization](#get-all-members-of-organization)
60+
* [Get members of organization with 2FA disabled](#get-members-of-organization-with-2fa-disabled)
61+
* [Get members of organization with "company" field undefined](#get-members-of-organization-with-company-field-undefined)
62+
* [Get projects which don't have any assigned team](#get-projects-which-dont-have-any-assigned-team)
63+
* [Get users which have undefined or hidden email](#get-users-which-have-undefined-or-hidden-email)
64+
* [Get users which may have not suitable fullname](#get-users-which-maye-have-not-suitable-fullname)
65+
* [Get repositories with undefined licenses](#get-repositories-with-undefined-licenses)
66+
* [Get repositories which seems to be unconform (i.e. missing files)](#get-repositories-which-seems-to-be-unconform-ie-missing-files)
67+
* [Get repositories which seems to be empty or have not enough files](#get-repositories-which-seems-to-be-empty-or-have-not-enough-files)
68+
* [Define users permissions for all projects to "push"](#define-users-permissions-for-all-projects-to-push)
69+
* [Define teams permissions for all projects to "push"](#define-teams-permissions-for-all-projects-to-push)
70+
* [Play with GitHub CLI (GH)](#play-with-github-cli-gh)
71+
* [Prerequisites](#prerequisites-1)
72+
* [Prepare project](#prepare-project-1)
73+
* [Third party elements](#third-party-elements-1)
74+
* [Features](#features-2)
75+
* [Make a backup of organization repositories](#make-a-backup-of-organization-repositories)
76+
* [Check if there are vulnerabilities alerts in organisation repositories](#check-if-there-are-vulnerabilities-alerts-in-organisation-repositories)
77+
* [Check if there are leaks in organisation repositories (using gitleaks)](#check-if-there-are-leaks-in-organisation-repositories-using-gitleaks)
78+
* [Play with GitLab web API](#play-with-gitlab-web-api)
79+
* [Prerequisites](#prerequisites-2)
80+
* [Prepare projects](#prepare-project-2)
81+
* [Features](#features-3)
82+
* [Make a backup of organization repositories](#make-a-backup-of-organization-repositories-1)
83+
* [Check if there are leaks in organisation repositories (using gitleaks)](#check-if-there-are-leaks-in-organisation-repositories-using-gitleaks-1)
84+
* [Licenses inventory](#licenses-inventory)
85+
* [Disclaimer](#disclaimer)
86+
* [Prerequisites](#prerequisites-3)
87+
* [Fill the configuration file](#fill-the-configuration-file)
88+
* [Run the tool](#run-the-tool)
89+
* [Run the tests](#run-the-tests)
90+
* [Managed platforms and environments](#managed-platforms)
91+
* [Go with go.mod](#go-language)
92+
* [Gradle with build.gradle(.kts)](#gradle-environment)
93+
* [Rust with Cargo.lock](#rust-environment)
94+
* [JavaScript / Node.js with package.json](#javascriptnodejs-environment)
95+
* [Swift with Package.swift](#swift--spm-environment)
96+
* [Dart / Flutter with pubspec.yaml](#dart--flutter-environment)
97+
* [Notes](#notes)
98+
3799
# The "diver" of source code and commits
38100

39101
## File of words
@@ -404,7 +466,7 @@ Then the Shell script will pick configuration details from the Ruby configuratio
404466

405467
The Python code will process JSON data, the Shell script will previously make a CURL request to to GraphQL API.
406468

407-
You need to define in the _configuration.rb_ files the Github organisation at **GITHUB_ORGANIZATION_NAME** and also your GitHub personal token at ** GITHUB_PERSONAL_ACCESS_TOKEN**.
469+
You need to define in the _configuration.rb_ files the Github organisation at **GITHUB_ORGANIZATION_NAME** and also your GitHub personal token at **GITHUB_PERSONAL_ACCESS_TOKEN**.
408470

409471
**You should also have your _git_ environment ready i.e. add your SSH private key if you clone by SSH for example. _gh_ must be installed, and python3 be ready.**
410472

@@ -430,7 +492,7 @@ The [gitleaks](https://github.com/zricethezav/gitleaks) tool will be used to loo
430492
brew install gitleaks
431493
```
432494

433-
You need to define in the _configuration.rb_ files the Github organisation at **GITHUB_ORGANIZATION_NAME** and also your GitHub personal token at ** GITHUB_PERSONAL_ACCESS_TOKEN**.
495+
You need to define in the _configuration.rb_ files the Github organisation at **GITHUB_ORGANIZATION_NAME** and also your GitHub personal token at **GITHUB_PERSONAL_ACCESS_TOKEN**.
434496

435497
**You should also have your _git_ environment ready i.e. add your SSH private key if you clone by SSH for example. _gh_ must be installed, and _python3_ be ready. Obviously _gitleaks_ must be installed**
436498

Lines changed: 203 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,203 @@
1+
#!/bin/bash
2+
# Software Name: floss-toolbox
3+
# SPDX-FileCopyrightText: Copyright (c) 2020-2023 Orange
4+
# SPDX-License-Identifier: Apache-2.0
5+
#
6+
# This software is distributed under the Apache 2.0 license.
7+
#
8+
# Author: Pierre-Yves LAPERSONNE <pierreyves(dot)lapersonne(at)orange(dot)com> et al.
9+
10+
# Since...............: 10/03/2023
11+
# Description.........: Make a dry-run of the LicensesIventory module to check if everything is ready to use
12+
# Version.............: 1.0.0
13+
14+
set -eu
15+
16+
# Couts
17+
# -----
18+
19+
NUMBER_OF_CHECKS=0
20+
NUMBER_OF_SUCCESS=0
21+
NUMBER_OF_WARNINGS=0
22+
NUMBER_OF_ERRORS=0
23+
24+
# Utils
25+
# -----
26+
27+
# $1 - File name to test
28+
CheckIfFileExists(){
29+
if [ ! -f "$1" ]; then
30+
echo "⛔ ERROR: The file '$1' does not exist"
31+
NUMBER_OF_ERRORS=$((NUMBER_OF_ERRORS+1))
32+
else
33+
echo "✅ Cool! The file '$1' exists"
34+
NUMBER_OF_SUCCESS=$((NUMBER_OF_SUCCESS+1))
35+
fi
36+
NUMBER_OF_CHECKS=$((NUMBER_OF_CHECKS+1))
37+
}
38+
39+
# $1 - FiDirectory name to test
40+
CheckIfDirectoryExists(){
41+
if [ ! -d "$1" ]; then
42+
echo "⛔ ERROR: The directory '$1' does not exist"
43+
NUMBER_OF_ERRORS=$((NUMBER_OF_ERRORS+1))
44+
else
45+
echo "✅ Cool! The directory '$1' exists"
46+
NUMBER_OF_SUCCESS=$((NUMBER_OF_SUCCESS+1))
47+
fi
48+
NUMBER_OF_CHECKS=$((NUMBER_OF_CHECKS+1))
49+
}
50+
51+
# $1 - Runtime name
52+
# $2 - Command to check runtime
53+
# $3 - Expected / suggested version
54+
CheckIfRuntimeExists(){
55+
runtime_version=`$2`
56+
if [ $? != "0" ]; then # Exist status != 0 so runtime version check failed ; we assume the runtime is missing
57+
echo "❌ WARNING: It seems '$1' is not ready"
58+
NUMBER_OF_ERRORS=$((NUMBER_OF_ERRORS+1))
59+
else
60+
echo "✅ Cool! '$1' is available"
61+
echo -e "\t🔎 You should check if the version is at least '$3'"
62+
NUMBER_OF_SUCCESS=$((NUMBER_OF_SUCCESS+1))
63+
fi
64+
NUMBER_OF_CHECKS=$((NUMBER_OF_CHECKS+1))
65+
}
66+
67+
# $1 - Ruby configuration file
68+
# $2 - Key to test
69+
CheckIfConfigurationKeyDefined(){
70+
CONFIG_KEY=`cat $1 | grep $2 | cut -d= -f2 | tr -d '"' | sed 's/ //g'`
71+
if [ "$CONFIG_KEY" == "" ]; then
72+
echo "❔ WARNING: It seems '$2' is not defined in $1"
73+
NUMBER_OF_WARNINGS=$((NUMBER_OF_WARNINGS+1))
74+
else
75+
echo "✅ Cool! '$2' is defined"
76+
NUMBER_OF_SUCCESS=$((NUMBER_OF_SUCCESS+1))
77+
fi
78+
NUMBER_OF_CHECKS=$((NUMBER_OF_CHECKS+1))
79+
}
80+
81+
# $1 - The name of the Python module to test
82+
CheckIfPythonModuleInstalled(){
83+
if [ "$#" -ne 1 ]; then
84+
echo "❌ WARNING: It seems '$1' is not ready"
85+
NUMBER_OF_ERRORS=$((NUMBER_OF_ERRORS+1))
86+
else
87+
python3 -c "import $1"
88+
result=$?
89+
if [ $result == "0" ]; then
90+
echo "✅ Cool! Python module '$1' is available"
91+
NUMBER_OF_SUCCESS=$((NUMBER_OF_SUCCESS+1))
92+
else
93+
echo "❌ WARNING: It seems Python modyle '$1' is not installed"
94+
NUMBER_OF_ERRORS=$((NUMBER_OF_ERRORS+1))
95+
fi
96+
fi
97+
NUMBER_OF_CHECKS=$((NUMBER_OF_CHECKS+1))
98+
}
99+
# Licenses Inventory tool
100+
# -----------------------
101+
102+
echo -e "\n------------------------------------------"
103+
echo "Assertions for the Licenses Inventory tool"
104+
echo "------------------------------------------"
105+
106+
echo -e "\nCheck if main folders exist..."
107+
CheckIfDirectoryExists "./sources"
108+
CheckIfDirectoryExists "./tests"
109+
110+
echo -e "\nCheck sources files..."
111+
CheckIfFileExists "./sources/common/__init__.py"
112+
CheckIfFileExists "./sources/common/datas.py"
113+
CheckIfFileExists "./sources/common/files.py"
114+
CheckIfFileExists "./sources/common/filters.py"
115+
CheckIfFileExists "./sources/common/names.py"
116+
CheckIfFileExists "./sources/configuration/__init__.py"
117+
CheckIfFileExists "./sources/configuration/config.py"
118+
CheckIfFileExists "./sources/dependencies/__init__.py"
119+
CheckIfFileExists "./sources/dependencies/dependencies.py"
120+
CheckIfFileExists "./sources/dependencies/parsings.py"
121+
CheckIfFileExists "./sources/search/__init__.py"
122+
CheckIfFileExists "./sources/search/downloads.py"
123+
CheckIfFileExists "./sources/search/parsings.py"
124+
CheckIfFileExists "./sources/search/search.py"
125+
CheckIfFileExists "./sources/__init__.py"
126+
CheckIfFileExists "./sources/main.py"
127+
CheckIfFileExists "./config.ini"
128+
129+
echo -e "\nCheck integration test files..."
130+
CheckIfFileExists "./tests/integrationtests/data/gradle/dependency_github.gradle"
131+
CheckIfFileExists "./tests/integrationtests/data/gradle/dependency_maven_central.gradle"
132+
CheckIfFileExists "./tests/integrationtests/data/gradle/license_github.json"
133+
CheckIfFileExists "./tests/integrationtests/data/gradle/license_maven_central.pom"
134+
CheckIfFileExists "./tests/integrationtests/data/gradle/version_maven_central.json"
135+
CheckIfFileExists "./tests/integrationtests/data/package_json/license_package_json.html"
136+
CheckIfFileExists "./tests/integrationtests/data/package_json/package.json"
137+
CheckIfFileExists "./tests/integrationtests/data/roast/Cargo.lock"
138+
CheckIfFileExists "./tests/integrationtests/data/config.ini"
139+
CheckIfFileExists "./tests/integrationtests/test_search.py"
140+
141+
echo -e "\nCheck unit test files..."
142+
CheckIfFileExists "./tests/unittests/data/config/config_no_data.ini"
143+
CheckIfFileExists "./tests/unittests/data/config/config.ini"
144+
CheckIfFileExists "./tests/unittests/data/get_content_by_name/my_gradle_file.txt"
145+
CheckIfFileExists "./tests/unittests/data/get_content_by_name/package.json"
146+
CheckIfFileExists "./tests/unittests/data/gradle/license_github.json"
147+
CheckIfFileExists "./tests/unittests/data/gradle/license_maven_central.pom"
148+
CheckIfFileExists "./tests/unittests/data/gradle/version.json"
149+
CheckIfFileExists "./tests/unittests/data/package_json/license_package_json.html"
150+
CheckIfFileExists "./tests/unittests/data/roast/license_roast.json"
151+
CheckIfFileExists "./tests/unittests/data/dependency_a.txt"
152+
CheckIfFileExists "./tests/unittests/data/dependency_b.txt"
153+
CheckIfFileExists "./tests/unittests/data/filename_by_name.test"
154+
CheckIfFileExists "./tests/unittests/data/files_read.txt"
155+
CheckIfFileExists "./tests/unittests/test_config.py"
156+
CheckIfFileExists "./tests/unittests/test_dependency.py"
157+
CheckIfFileExists "./tests/unittests/test_files_check_the_directory.py"
158+
CheckIfFileExists "./tests/unittests/test_files_get_the_filenames_by_name.py"
159+
CheckIfFileExists "./tests/unittests/test_files_write_and_read.py"
160+
CheckIfFileExists "./tests/unittests/test_filter.py"
161+
CheckIfFileExists "./tests/unittests/test_parsing_download.py"
162+
CheckIfFileExists "./tests/unittests/test_parsing.py"
163+
164+
# Runtimes and tools
165+
# ------------------
166+
167+
echo -e "\n-----------------------"
168+
echo "Assertions for runtimes"
169+
echo "-----------------------"
170+
171+
echo -e "\nCheck for Python3..."
172+
CheckIfRuntimeExists "Python3" "python3 --version" "3.8.5"
173+
174+
echo -e "\nCheck for Python modules"
175+
CheckIfPythonModuleInstalled "requests"
176+
CheckIfPythonModuleInstalled "xmltodict"
177+
CheckIfPythonModuleInstalled "pytest"
178+
179+
# Units tests
180+
# -----------
181+
182+
echo -e "\n----------------------------------"
183+
echo "Run of LicensesInventory unit test"
184+
echo "----------------------------------"
185+
186+
echo -e "\nRunning integration tests..."
187+
python3 -m pytest ./tests/integrationtests/*.py
188+
189+
echo -e "\nRunning unit tests..."
190+
python3 -m pytest ./tests/unittests/*.py
191+
192+
# Conclusion
193+
# ----------
194+
195+
echo -e "\n----------"
196+
echo "Conclusion"
197+
echo "----------"
198+
199+
echo -e "\nDry-run done! See the logs above to check all points controls."
200+
echo -e "\tNumber of controls.......: $NUMBER_OF_CHECKS"
201+
echo -e "\tNumber of success........: $NUMBER_OF_SUCCESS"
202+
echo -e "\tNumber of warnings.......: $NUMBER_OF_WARNINGS"
203+
echo -e "\tNumber of errors.........: $NUMBER_OF_ERRORS"

toolbox/LicensesInventory/sources/__init__.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
#!/usr/bin/env python3
22
# -*- coding: utf-8 -*-
33
# Software Name: floss-toolbox
4-
# SPDX-FileCopyrightText: Copyright (c) 2023 Orange
4+
# SPDX-FileCopyrightText: Copyright (c) 2020-2023 Orange
55
# SPDX-License-Identifier: Apache-2.0
66
#
77
# This software is distributed under the Apache 2.0 license.

toolbox/LicensesInventory/sources/common/__init__.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
#!/usr/bin/env python3
22
# -*- coding: utf-8 -*-
33
# Software Name: floss-toolbox
4-
# SPDX-FileCopyrightText: Copyright (c) 2023 Orange
4+
# SPDX-FileCopyrightText: Copyright (c) 2020-2023 Orange
55
# SPDX-License-Identifier: Apache-2.0
66
#
77
# This software is distributed under the Apache 2.0 license.

toolbox/LicensesInventory/sources/common/datas.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
#!/usr/bin/env python3
22
# -*- coding: utf-8 -*-
33
# Software Name: floss-toolbox
4-
# SPDX-FileCopyrightText: Copyright (c) 2023 Orange
4+
# SPDX-FileCopyrightText: Copyright (c) 2020-2023 Orange
55
# SPDX-License-Identifier: Apache-2.0
66
#
77
# This software is distributed under the Apache 2.0 license.

toolbox/LicensesInventory/sources/common/files.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
#!/usr/bin/env python3
22
# -*- coding: utf-8 -*-
33
# Software Name: floss-toolbox
4-
# SPDX-FileCopyrightText: Copyright (c) 2023 Orange
4+
# SPDX-FileCopyrightText: Copyright (c) 2020-2023 Orange
55
# SPDX-License-Identifier: Apache-2.0
66
#
77
# This software is distributed under the Apache 2.0 license.

toolbox/LicensesInventory/sources/common/names.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
#!/usr/bin/env python3
22
# -*- coding: utf-8 -*-
33
# Software Name: floss-toolbox
4-
# SPDX-FileCopyrightText: Copyright (c) 2023 Orange
4+
# SPDX-FileCopyrightText: Copyright (c) 2020-2023 Orange
55
# SPDX-License-Identifier: Apache-2.0
66
#
77
# This software is distributed under the Apache 2.0 license.

toolbox/LicensesInventory/sources/configuration/__init__.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
#!/usr/bin/env python3
22
# -*- coding: utf-8 -*-
33
# Software Name: floss-toolbox
4-
# SPDX-FileCopyrightText: Copyright (c) 2023 Orange
4+
# SPDX-FileCopyrightText: Copyright (c) 2020-2023 Orange
55
# SPDX-License-Identifier: Apache-2.0
66
#
77
# This software is distributed under the Apache 2.0 license.

toolbox/LicensesInventory/sources/configuration/config.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
#!/usr/bin/env python3
22
# -*- coding: utf-8 -*-
33
# Software Name: floss-toolbox
4-
# SPDX-FileCopyrightText: Copyright (c) 2023 Orange
4+
# SPDX-FileCopyrightText: Copyright (c) 2020-2023 Orange
55
# SPDX-License-Identifier: Apache-2.0
66
#
77
# This software is distributed under the Apache 2.0 license.

0 commit comments

Comments
 (0)