Open-set recognition is the problem that matters most in deployed malware classification. A model that can confidently label samples from known families and reject anything it has never seen is more useful than one that simply assigns every input to the closest known category. In the domain of graph-based malware analysis, this problem is particularly acute: Function Call Graphs encode the structural behavior of programs, and families evolve or emerge faster than labels can be curated. A new paper by Junru Zhu, Yixin Yang, Xiaoqing Ding, and Ruoyu Qi tests a seemingly natural way to improve open-set rejection on graph data: enrich graph prototypes with community structure detected by the Louvain algorithm. The result is a rigorous negative finding that carries implications for how the entire field evaluates graph-based open-set classifiers.
The Open-Set Problem in Graph Malware Classification
Traditional supervised malware classifiers assume that every test sample belongs to a family seen during training. In practice, that assumption fails constantly. New malware families appear, existing ones mutate, and packers or obfuscators shift structural signatures. Open-set recognition addresses this by training a model to separate known families from an unknown class, either by assigning a rejection threshold or by ranking samples against a known-class boundary.
Graph neural networks, and specifically Graph Isomorphism Networks (GIN), have become a standard backbone for malware classification because they can distinguish graph structures at a level of granularity that matches the complexity of function call graphs. The FCG-MFD benchmark, introduced by Hadi et al. in 2025, provides a large-scale corpus of function call graphs extracted from real malware samples across multiple repositories, with metadata features attached to each node. The dataset has been widely adopted as a testbed for graph-based malware family detection, and it is the corpus used in this study.
The core question the paper asks is narrow but consequential: does adding community-level summary information improve a graph model's ability to reject unseen malware families? This is not a theoretical curiosity. Community structure in a function call graph can reveal modular organizational patterns: groups of functions that interact tightly with each other but loosely with the rest of the program. If a malware family tends to organize its function calls in a particular community pattern, then summarizing that structure could help distinguish it from both other known families and unknown ones.
What the Paper Tests
The experimental design is straightforward in concept but careful in execution. The authors take the deduplicated, conflict-audited version of the FCG-MFD corpus and train a GIN model on a subset of known malware families. For each family, they compute nearest-prototype embeddings: the centroid of the known-family samples in the graph embedding space serves as a representative point, and classification is done by assigning each sample to the nearest prototype.
The innovation is in what gets fed into the embedding. The baseline is the raw GIN embedding plus a set of dimension-matched generic topology features: standard graph statistics like node count, edge count, and degree distributions that describe the overall shape of the graph without reference to any community structure. The experimental condition adds Louvain-community summaries: features computed from the partition that the Louvain algorithm finds in the function call graph, such as community counts, modularity scores, and aggregate statistics per community. These community features are then residualized against the generic topology using only known-family training data. This residualization is a critical design choice. It strips away any variation in community features that is already captured by the generic topology, leaving only the community signal that is genuinely orthogonal to raw graph size and shape.
Three optimization seeds and five held-out malware families were used. The held-out families are not present in the training set at all, so any correct rejection of them constitutes a genuine open-set result. Nearest-prototype scoring is applied in both the baseline and community-enriched embedding spaces, and the results are compared across several metrics: ranking performance, false-positive rate at 95 percent unknown recall, accepted-known macro F1, and the Open-Set Classification Rate (OSCR).
Why Residualization Matters
The residualization step deserves attention because it addresses a confound that is easy to overlook. Community structure in a graph is not independent of graph size and density. Larger graphs tend to have more communities, and denser graphs tend to have tighter community boundaries. If you add raw Louvain features to a graph embedding without accounting for these effects, you are partly measuring graph scale rather than community organization. The generic topology features already capture graph scale, so residualizing the community features against them isolates the incremental community signal.
In this study, the residualized community features still retain a link to graph scale. That is, even after the residualization step, the community signal is not fully orthogonal to the size and density of the graph. This finding itself is informative: it suggests that Louvain communities in function call graphs are not purely modular organizational units that carry independent structural information. They are partly artifacts of how large and dense the underlying graph is.
The Results: A Clean Negative Finding
The headline result is that residual community features do not produce stable held-out-family rejection. Across the five held-out families and three optimization seeds, the community-enriched prototypes fail to consistently separate unknown families from known ones. More troubling than the average failure is the pattern of the failures.
Ranking effects reverse across families. For some held-out families, the community-enriched embedding improves the ranking of unknown samples relative to known ones. For other families, it makes the ranking worse. The direction of the effect is not consistent, which means the community signal is not a systematic improvement to the geometry of the embedding space. It is noisy, and its noise correlates with family-specific properties that the model cannot generalize across.
The false-positive rate at 95 percent unknown recall worsens for every single held-out family. This is the most damning result. If the goal is to reject unknown families while keeping known-family false positives low, adding community features makes the situation worse on all five families simultaneously. A validation-fitted threshold on the community-enriched space rejects only 4.48 percent of unknown samples, which is a very low rejection rate for an open-set system that is supposed to be catching novel threats.
There is one positive signal: accepted-known macro F1 improves in every family. The community features help the model better classify samples that belong to families it has already seen. But this comes at a cost. The geometry of the known-class clusters changes in a way that moves them closer together relative to the unknown region, rather than pushing the unknown region away. The community features reshape known-class separation without creating a stable unknown margin.
Statistical rigor matters here. With five independent family units, the smallest attainable exact two-sided sign-flip p-value is 0.0625. The authors report this value, which means that even if the observed improvements in known-family F1 were perfectly consistent across all five families, the result would not reach conventional significance at the 0.05 level given the combinatorial constraint of five binary outcomes. The paper is honest about this ceiling.
What Outperforms Community Features
The paper's comparison against simple classifier uncertainty is revealing. Classifier uncertainty, in the form of how confidently the GIN assigns a sample to a known prototype, performs better on ranking, high-recall rejection, and OSCR than the community-enriched approach. This is not because uncertainty is theoretically superior in all settings; it is because uncertainty captures a property of the embedding that is directly relevant to the open-set task: how far a sample sits from the known-class boundary in the learned space.
The practical takeaway is that, in this GIN/FCG-MFD setting, the marginal value of community structure for open-set rejection is negligible or negative. The graph embedding itself, combined with uncertainty-based scoring, already captures enough structural information to define a useful decision boundary. Adding Louvain communities on top does not help, and in some configurations actively degrades the boundary.
Limitations and Honest Inferences
The paper is transparent about its constraints. Five held-out families and three seeds give the study limited statistical power. The sign-flip p-value ceiling of 0.0625 means that the statistical test cannot formally reject the null hypothesis of no effect, even though the observed direction of the effect is consistently negative for the most important metrics. This is not a failure of the experimental design; it is a honest acknowledgment of the combinatorial difficulty of significance testing with a small number of independent units.
The deduplicated, conflict-audited nature of the corpus is a strength, but it also means the results are specific to the FCG-MFD family of datasets and the particular malware families contained within them. Different graph construction methods, different family taxonomies, or different malware domains could yield different conclusions. The paper does not claim that Louvain communities are universally useless for open-set graph classification; it claims they do not help in this specific setting.
One inference that can be drawn more broadly is about the nature of community structure in function call graphs. If residualized community features remain associated with graph scale even after controlling for topology, then Louvain communities in this domain are partly confounded with graph size. This suggests that for malware function call graphs, community detection may not be capturing a semantically meaningful modular decomposition of the program's structure, but rather reflecting the sheer scale and complexity of the call graph.
What This Means for Practitioners
For a working developer or security engineer evaluating graph-based malware classifiers, the practical implication is clear: do not assume that adding community-detection features will improve open-set rejection. The paper's findings suggest that, at least for function call graph-based malware analysis with GIN encoders, simpler approaches based on classifier uncertainty are more reliable for high-recall rejection tasks.
This does not mean that community structure is irrelevant in all graph-based security applications. For tasks like anomaly detection in network traffic graphs or community-aware threat clustering, Louvain-based features may still carry independent signal. The paper's finding is specific to the combination of malware family classification, nearest-prototype scoring, and the FCG-MFD corpus.
The broader methodological recommendation is about evaluation rigor. The paper argues that graph open-set evaluations should always pair structural features with matched topology controls, operational thresholds calibrated on validation data, and held-out-family analysis rather than aggregate metrics alone. Without these controls, it is easy to mistake a scaling artifact for a structural signal. This checklist is directly applicable to any researcher working on graph-based open-set problems in cybersecurity or beyond.
Where the Field Goes From Here
The negative result opens several productive questions. Why does community structure help known-family classification but hurt open-set rejection? One plausible mechanism is that community features refine the internal geometry of known-class clusters without expanding the separation between known and unknown regions. This is a common failure mode in open-set learning: the model becomes better at distinguishing what it already knows without learning what it does not know.
Alternative community detection methods, different graph neural network architectures, or different residualization strategies could change the outcome. The paper uses Louvain because it is fast and well-established, but other community detection algorithms might capture modular structure in a way that is more orthogonal to graph scale. Similarly, using a graph encoder other than GIN might produce embeddings where community structure carries more independent information for open-set tasks.
For now, the paper provides a clear and well-justified finding: in the specific setting of GIN-based malware family recognition on the FCG-MFD corpus, residual Louvain community features do not create a stable unknown margin. They change known-class geometry. They worsen high-recall rejection. And they do so consistently enough that the burden of proof now falls on anyone who wants to claim that community structure adds value for open-set graph malware classification.