forked from apache/atlas
-
Notifications
You must be signed in to change notification settings - Fork 9
Tags intg test #5438
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
sriram-atlan
wants to merge
20
commits into
staging
Choose a base branch
from
tags_intg_test
base: staging
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Tags intg test #5438
Changes from all commits
Commits
Show all changes
20 commits
Select commit
Hold shift + click to select a range
2cb830d
integration test setup (wip)
sriram-atlan eda9ae9
initial work for e2e tests (wip)
sriram-atlan 2c9574b
keep only required files. use spring profiles (wip)
sriram-atlan fab868a
Change version for containers
nikhilbonte21 68006a3
Fix active profiles
nikhilbonte21 c4f4086
e2e test for create asset and kafka notification verification
sriram-atlan 87f83ea
run intg tests from maven
sriram-atlan 29512f5
run tests after running install
sriram-atlan ffb5bf7
run integration tests as separate step
sriram-atlan 7dcdd0d
fix syntax for docker options
sriram-atlan 02c1549
build docker image with shell before running integration tests
sriram-atlan 188ef79
remove double builds
sriram-atlan 8ffe8ac
remove double builds
sriram-atlan b045735
remove install
sriram-atlan db07ae5
remove grafana dashboard and update shell script name
sriram-atlan 6030d9d
replace package with install
sriram-atlan 5dc1708
remove DinD and add verify docker as a step
sriram-atlan 12a95ab
use settings that i dont understand
sriram-atlan ceac6d5
list docker images
sriram-atlan 94f19ff
build and load docker image for atlas
sriram-atlan File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
6 changes: 4 additions & 2 deletions
6
common/src/main/java/org/apache/atlas/service/redis/RedisServiceImpl.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
6 changes: 4 additions & 2 deletions
6
common/src/main/java/org/apache/atlas/service/redis/RedisServiceLocalImpl.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
77 changes: 77 additions & 0 deletions
77
repository/src/test/java/org/apache/atlas/repository/integration/TestEntityStreamUtil.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,77 @@ | ||
package org.apache.atlas.repository.integration; | ||
|
||
import com.fasterxml.jackson.databind.JsonNode; | ||
import com.fasterxml.jackson.databind.ObjectMapper; | ||
import com.fasterxml.jackson.databind.node.ArrayNode; | ||
import org.apache.atlas.model.instance.AtlasEntity; | ||
import org.apache.atlas.model.instance.AtlasEntityHeader; | ||
import org.apache.atlas.model.instance.AtlasEntityHeaders; | ||
import org.apache.atlas.model.instance.EntityMutationResponse; | ||
import org.apache.atlas.repository.store.graph.v2.AtlasEntityStream; | ||
import org.apache.atlas.repository.store.graph.v2.EntityStream; | ||
import org.apache.atlas.type.AtlasTypeRegistry; | ||
|
||
import java.io.IOException; | ||
import java.util.ArrayList; | ||
import java.util.HashMap; | ||
import java.util.List; | ||
import java.util.Map; | ||
|
||
public class TestEntityStreamUtil { | ||
private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper(); | ||
|
||
public static class EntityCreationRequest { | ||
private final String jsonRequest; | ||
private List<AtlasEntity> entities; | ||
private Map<String, AtlasEntityHeader> guidHeaderMap; | ||
|
||
public EntityCreationRequest(String jsonRequest) { | ||
this.jsonRequest = jsonRequest; | ||
this.entities = new ArrayList<>(); | ||
this.guidHeaderMap = new HashMap<>(); | ||
} | ||
|
||
public EntityStream toEntityStream(AtlasTypeRegistry typeRegistry) throws IOException { | ||
JsonNode root = OBJECT_MAPPER.readTree(jsonRequest); | ||
ArrayNode entitiesNode = (ArrayNode) root.get("entities"); | ||
|
||
for (JsonNode entityNode : entitiesNode) { | ||
AtlasEntity.AtlasEntityWithExtInfo entityWithExtInfo = new AtlasEntity.AtlasEntityWithExtInfo(); | ||
AtlasEntity entity = new AtlasEntity(); | ||
|
||
entity.setTypeName(entityNode.get("typeName").asText()); | ||
|
||
// Handle attributes | ||
JsonNode attributesNode = entityNode.get("attributes"); | ||
Map<String, Object> attributes = new HashMap<>(); | ||
attributesNode.fields().forEachRemaining(field -> | ||
attributes.put(field.getKey(), field.getValue().asText()) | ||
); | ||
entity.setAttributes(attributes); | ||
|
||
entityWithExtInfo.setEntity(entity); | ||
entities.add(entity); | ||
} | ||
|
||
return new AtlasEntityStream(entities); | ||
} | ||
|
||
public void updateWithMutationResponse(EntityMutationResponse response) { | ||
// Update entities with assigned GUIDs | ||
for (AtlasEntityHeader header : response.getCreatedEntities()) { | ||
guidHeaderMap.put(header.getGuid(), header); | ||
} | ||
} | ||
|
||
public AtlasEntityHeaders toEntityHeaders() { | ||
AtlasEntityHeaders headers = new AtlasEntityHeaders(); | ||
headers.setGuidHeaderMap(guidHeaderMap); | ||
return headers; | ||
} | ||
} | ||
|
||
public static EntityCreationRequest fromJson(String json) { | ||
return new EntityCreationRequest(json); | ||
} | ||
|
||
} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,124 @@ | ||
#!/bin/bash | ||
# run-integration-tests.sh | ||
|
||
set -e | ||
|
||
echo "============================================" | ||
echo "Atlas Integration Tests Runner" | ||
echo "============================================" | ||
|
||
# Colors for output | ||
RED='\033[0;31m' | ||
GREEN='\033[0;32m' | ||
YELLOW='\033[1;33m' | ||
NC='\033[0m' # No Color | ||
export JAVA_HOME=`/usr/libexec/java_home -v 17` # Set Java 17 as JAVA_HOME | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. |
||
# Check if Docker is running | ||
if ! docker info > /dev/null 2>&1; then | ||
echo -e "${RED}Docker is not running. Please start Docker first.${NC}" | ||
exit 1 | ||
fi | ||
|
||
# Parse arguments | ||
SKIP_BUILD=false | ||
KEEP_CONTAINERS=false | ||
DEBUG=false | ||
|
||
while [[ "$#" -gt 0 ]]; do | ||
case $1 in | ||
--skip-build) SKIP_BUILD=true ;; | ||
--keep-containers) KEEP_CONTAINERS=true ;; | ||
--debug) DEBUG=true ;; | ||
-h|--help) | ||
echo "Usage: $0 [options]" | ||
echo "Options:" | ||
echo " --skip-build Skip building Atlas WAR and Docker image" | ||
echo " --keep-containers Keep containers running after tests" | ||
echo " --debug Enable debug logging" | ||
echo " -h, --help Show this help message" | ||
exit 0 | ||
;; | ||
*) echo "Unknown parameter: $1"; exit 1 ;; | ||
esac | ||
shift | ||
done | ||
|
||
# Step 1: Build Docker image | ||
echo -e "${YELLOW}Building Atlas Docker image...${NC}" | ||
docker buildx build --load -t atlanhq/atlas:test . | ||
|
||
if [ $? -ne 0 ]; then | ||
echo -e "${RED}Failed to build Docker image${NC}" | ||
exit 1 | ||
fi | ||
echo -e "${GREEN}Docker image built successfully${NC}" | ||
|
||
# Step 2: Clean up any existing test containers | ||
echo -e "${YELLOW}Cleaning up existing test containers...${NC}" | ||
docker rm -f atlas-test-zookeeper atlas-test-kafka atlas-test-cassandra \ | ||
atlas-test-elasticsearch atlas-test-redis atlas-test-atlas 2>/dev/null || true | ||
|
||
# Step 3: Set test properties | ||
export TESTCONTAINERS_REUSE_ENABLE=true | ||
export TESTCONTAINERS_RYUK_DISABLED=$KEEP_CONTAINERS | ||
|
||
if [ "$DEBUG" = true ]; then | ||
export TESTCONTAINERS_DEBUG=true | ||
MAVEN_OPTS="-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005" | ||
fi | ||
|
||
echo "Listing docker images" | ||
docker image ls | ||
|
||
# Step 4: Run integration tests | ||
echo -e "${YELLOW}Running integration tests...${NC}" | ||
|
||
if [ "$DEBUG" = true ]; then | ||
mvn test -pl webapp -Dtest=AtlasDockerIntegrationTest \ | ||
-Dorg.slf4j.simpleLogger.defaultLogLevel=debug \ | ||
-Dorg.testcontainers.log.level=DEBUG -Dsurefire.useFile=false | ||
else | ||
mvn test -pl webapp -Dtest=AtlasDockerIntegrationTest -Dsurefire.useFile=false | ||
fi | ||
|
||
TEST_RESULT=$? | ||
|
||
# Step 5: Collect logs if tests failed | ||
if [ $TEST_RESULT -ne 0 ]; then | ||
echo -e "${RED}Tests failed! Collecting logs...${NC}" | ||
|
||
mkdir -p target/test-logs | ||
|
||
# Get container logs | ||
for container in $(docker ps -a --filter "name=atlas-test" --format "{{.Names}}"); do | ||
echo "Collecting logs from $container..." | ||
docker logs $container > "target/test-logs/${container}.log" 2>&1 | ||
done | ||
|
||
echo -e "${YELLOW}Logs saved to target/test-logs/${NC}" | ||
fi | ||
|
||
# Step 6: Clean up containers if not keeping them | ||
if [ "$KEEP_CONTAINERS" = false ]; then | ||
echo -e "${YELLOW}Cleaning up test containers...${NC}" | ||
docker rm -f $(docker ps -a --filter "name=atlas-test" --format "{{.Names}}") 2>/dev/null || true | ||
else | ||
echo -e "${YELLOW}Keeping containers running (--keep-containers flag set)${NC}" | ||
echo "You can connect to Atlas at: http://localhost:21000" | ||
echo "To stop containers manually, run:" | ||
echo " docker rm -f \$(docker ps -a --filter 'name=atlas-test' --format '{{.Names}}')" | ||
fi | ||
|
||
# Step 7: Report results | ||
if [ $TEST_RESULT -eq 0 ]; then | ||
echo -e "${GREEN}============================================${NC}" | ||
echo -e "${GREEN}Integration tests completed successfully!${NC}" | ||
echo -e "${GREEN}============================================${NC}" | ||
exit 0 | ||
else | ||
echo -e "${RED}============================================${NC}" | ||
echo -e "${RED}Integration tests failed!${NC}" | ||
echo -e "${RED}Check target/test-logs for details${NC}" | ||
echo -e "${RED}============================================${NC}" | ||
exit 1 | ||
fi |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Bug: Redis Sentinel Configuration Not Applied
The
checkSentinelsList
field is declared but not initialized from theatlas.redis.sentinel.check_list.enabled
configuration. This means it always defaults tofalse
when used ingetCacheImplConfig()
, preventing the intended configuration from taking effect.