Skip to content

Sync beta with master #5401

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Parent: Sync beta with master
merged 14 commits into from
Sep 19, 2025
Merged

Sync beta with master #5401

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
f070e24
MLH-1240 improve refresh logic, reduce retry count and wait time
sriram-atlan Sep 15, 2025
3a6b41a
MLH-1240 use version number to decide to invoke typedef cache refresh…
sriram-atlan Sep 15, 2025
41df3cd
MLH-1240 reload only business metadata
sriram-atlan Sep 15, 2025
7c3e04e
MLH-1240 increment internal cache version
sriram-atlan Sep 15, 2025
248ea12
MLH-1240 reload business and classification typedef
sriram-atlan Sep 15, 2025
28e84d9
MLH-1240 use consistent version and reload if necessary in all typede…
sriram-atlan Sep 15, 2025
1a6c6f0
MLH-1240 update cache when current version is less than redis version
sriram-atlan Sep 15, 2025
887ddc6
MLH-1240 refresh custom types always (wip testing)
sriram-atlan Sep 16, 2025
b42fdbf
MLH-1240 refresh enum and always read from transient registry
sriram-atlan Sep 16, 2025
cb8c0ab
MLH-1240 track version for all custom types to load them independently
sriram-atlan Sep 16, 2025
b91db52
MLH-1240 commit update
sriram-atlan Sep 16, 2025
ce513e4
MLH-1240 clear that type of atlas typedef and reload to handle delete…
sriram-atlan Sep 17, 2025
0cf1155
MLH-1240 add struct, enum and relationship def and refactor cache ref…
sriram-atlan Sep 17, 2025
7e06e58
Merge pull request #5380 from atlanhq/mlh-1240-custom-metadata-consis…
sriram-atlan Sep 18, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/maven.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ on:
- tagscanarymerge
- fixlabels
- interceptapis
- mlh-1240-custom-metadata-consistency

jobs:
build:
Expand Down
7 changes: 7 additions & 0 deletions common/src/main/java/org/apache/atlas/repository/Constants.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -538,6 +538,13 @@ public enum SupportedFileExtensions { XLSX, XLS, CSV }
add(STAKEHOLDER_TITLE_ENTITY_TYPE);
}};

public static final String TYPEDEF_ENUM_CACHE_LATEST_VERSION = "typdef.enum.cache.version";
public static final String TYPEDEF_BUSINESS_METADATA_CACHE_LATEST_VERSION = "typdef.bm.cache.version";
public static final String TYPEDEF_CLASSIFICATION_METADATA_CACHE_LATEST_VERSION = "typdef.cls.cache.version";
public static final String TYPEDEF_STRUCT_CACHE_LATEST_VERSION = "typdef.struct.cache.version";
public static final String TYPEDEF_ENTITY_CACHE_LATEST_VERSION = "typdef.entity.cache.version";
public static final String TYPEDEF_RELATIONSHIP_CACHE_LATEST_VERSION = "typdef.relationship.cache.version";

private Constants() {
}

Expand Down
17 changes: 17 additions & 0 deletions common/src/main/java/org/apache/atlas/service/redis/AbstractRedisService.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
import org.apache.atlas.service.metrics.MetricUtils;
import org.apache.commons.configuration.Configuration;
import org.apache.commons.lang.ArrayUtils;
import org.apache.commons.lang.StringUtils;
import org.redisson.api.RLock;
import org.redisson.api.RedissonClient;
import org.redisson.config.Config;
Expand Down Expand Up @@ -191,6 +192,22 @@ public String getValue(String key) {
}
}

@Override
public String getValue(String key, String defaultValue) {
try {
String value = getValue(key);
if (StringUtils.isEmpty(value)) {
return defaultValue;
} else {
return value;
}
} catch (Exception e) {
MetricUtils.recordRedisConnectionFailure();
getLogger().error("Redis getValue operation failed for key: {}", key, e);
throw e;
}
}

@Override
public String putValue(String key, String value) {
try {
Expand Down
2 changes: 2 additions & 0 deletions common/src/main/java/org/apache/atlas/service/redis/RedisService.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ public interface RedisService {

String getValue(String key);

String getValue(String key, String defaultValue);

String putValue(String key, String value);

String putValue(String key, String value, int timeout);
Expand Down
5 changes: 5 additions & 0 deletions common/src/main/java/org/apache/atlas/service/redis/RedisServiceLocalImpl.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,11 @@ public String getValue(String key) {
return super.getValue(key);
}

@Override
public String getValue(String key, String defaultValue) {
return null;
}

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: Redis Value Retrieval Ignores Default

The getValue(String key, String defaultValue) method in RedisServiceLocalImpl always returns null. This ignores the defaultValue parameter and violates the method's contract to provide a default when the key is not found or its value is empty, which can lead to NumberFormatException in callers.

Fix in Cursor Fix in Web

@Override
public String putValue(String key, String value, int timeout) {
return super.putValue(key, value, timeout);
Expand Down
323 changes: 167 additions & 156 deletions repository/src/main/java/org/apache/atlas/repository/graph/TypeCacheRefresher.java
View file
Open in desktop

Large diffs are not rendered by default.

65 changes: 64 additions & 1 deletion ...c/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@
import org.apache.atlas.repository.patches.SuperTypesUpdatePatch;
import org.apache.atlas.repository.patches.AtlasPatchManager;
import org.apache.atlas.repository.patches.AtlasPatchRegistry;
import org.apache.atlas.service.redis.RedisService;
import org.apache.atlas.store.AtlasTypeDefStore;
import org.apache.atlas.type.AtlasEntityType;
import org.apache.atlas.type.AtlasStructType.AtlasAttribute;
Expand Down Expand Up @@ -101,15 +102,23 @@ public class AtlasTypeDefStoreInitializer implements ActiveStateChangeHandler {
private final Configuration conf;
private final AtlasGraph graph;
private final AtlasPatchManager patchManager;
private final RedisService redisService;
private static long CURRENT_ENUM_TYPEDEF_INTERNAL_VERSION;
private static long CURRENT_BUSINESS_METADATA_TYPEDEF_INTERNAL_VERSION;
private static long CURRENT_CLASSIFICATION_TYPEDEF_INTERNAL_VERSION;
private static long CURRENT_STRUCT_TYPEDEF_INTERNAL_VERSION;
private static long CURRENT_ENTITY_TYPEDEF_INTERNAL_VERSION;
private static long CURRENT_RELATIONSHIP_TYPEDEF_INTERNAL_VERSION;

@Inject
public AtlasTypeDefStoreInitializer(AtlasTypeDefStore typeDefStore, AtlasTypeRegistry typeRegistry,
AtlasGraph graph, Configuration conf, AtlasPatchManager patchManager) throws AtlasBaseException {
AtlasGraph graph, Configuration conf, AtlasPatchManager patchManager, RedisService redisService) throws AtlasBaseException {
this.typeDefStore = typeDefStore;
this.typeRegistry = typeRegistry;
this.conf = conf;
this.graph = graph;
this.patchManager = patchManager;
this.redisService = redisService;
}

@PostConstruct
Expand All @@ -118,6 +127,12 @@ public void init() {

if (!HAConfiguration.isHAEnabled(conf)) {
startInternal();
CURRENT_ENUM_TYPEDEF_INTERNAL_VERSION = Long.parseLong(redisService.getValue(Constants.TYPEDEF_ENUM_CACHE_LATEST_VERSION, "1"));
CURRENT_BUSINESS_METADATA_TYPEDEF_INTERNAL_VERSION = Long.parseLong(redisService.getValue(Constants.TYPEDEF_BUSINESS_METADATA_CACHE_LATEST_VERSION, "1"));
CURRENT_CLASSIFICATION_TYPEDEF_INTERNAL_VERSION = Long.parseLong(redisService.getValue(Constants.TYPEDEF_CLASSIFICATION_METADATA_CACHE_LATEST_VERSION, "1"));
CURRENT_STRUCT_TYPEDEF_INTERNAL_VERSION = Long.parseLong(redisService.getValue(Constants.TYPEDEF_STRUCT_CACHE_LATEST_VERSION, "1"));
CURRENT_ENTITY_TYPEDEF_INTERNAL_VERSION = Long.parseLong(redisService.getValue(Constants.TYPEDEF_ENTITY_CACHE_LATEST_VERSION, "1"));
CURRENT_RELATIONSHIP_TYPEDEF_INTERNAL_VERSION = Long.parseLong(redisService.getValue(Constants.TYPEDEF_RELATIONSHIP_CACHE_LATEST_VERSION, "1"));
} else {
LOG.info("AtlasTypeDefStoreInitializer.init(): deferring type loading until instance activation");
}
Expand Down Expand Up @@ -407,6 +422,54 @@ public int getHandlerOrder() {
return HandlerOrder.TYPEDEF_STORE_INITIALIZER.getOrder();
}

public static long getCurrentEnumTypedefInternalVersion() {
return CURRENT_ENUM_TYPEDEF_INTERNAL_VERSION;
}

public static void setCurrentEnumTypedefInternalVersion(long version) {
CURRENT_ENUM_TYPEDEF_INTERNAL_VERSION = version;
}

public static long getCurrentBMTypedefInternalVersion() {
return CURRENT_BUSINESS_METADATA_TYPEDEF_INTERNAL_VERSION;
}

public static void setCurrentBMTypedefInternalVersion(long version) {
CURRENT_BUSINESS_METADATA_TYPEDEF_INTERNAL_VERSION = version;
}

public static long getCurrentClassificationTypedefInternalVersion() {
return CURRENT_CLASSIFICATION_TYPEDEF_INTERNAL_VERSION;
}

public static void setCurrentClassificationTypedefInternalVersion(long version) {
CURRENT_CLASSIFICATION_TYPEDEF_INTERNAL_VERSION = version;
}

public static long getCurrentStructTypedefInternalVersion() {
return CURRENT_STRUCT_TYPEDEF_INTERNAL_VERSION;
}

public static void setCurrentStructTypedefInternalVersion(long version) {
CURRENT_STRUCT_TYPEDEF_INTERNAL_VERSION = version;
}

public static long getCurrentEntityTypedefInternalVersion() {
return CURRENT_ENTITY_TYPEDEF_INTERNAL_VERSION;
}

public static void setCurrentEntityTypedefInternalVersion(long version) {
CURRENT_ENTITY_TYPEDEF_INTERNAL_VERSION = version;
}

public static long getCurrentRelationshipTypedefInternalVersion() {
return CURRENT_RELATIONSHIP_TYPEDEF_INTERNAL_VERSION;
}

public static void setCurrentRelationshipTypedefInternalVersion(long version) {
CURRENT_RELATIONSHIP_TYPEDEF_INTERNAL_VERSION = version;
}

private static boolean updateTypeAttributes(AtlasStructDef oldStructDef, AtlasStructDef newStructDef, boolean checkTypeVersion) {
boolean ret = isTypeUpdateApplicable(oldStructDef, newStructDef, checkTypeVersion);

Expand Down
Loading
Loading