Four decision rules visualized

Let me show you the core insight using a simple two-outcome example. Imagine we’re testing a treatment that targets:

• Cognitive function (outcome 1)
• Fatigue (outcome 2)

Each outcome has a treatment effect: the difference in success probability between the new treatment and control, and .

The four rules discussed here are:

1. Single Rule: improvement on prespecified (“primary”) outcome

• Tests if cognitive function improves:
• Ignores fatigue completely, so fatigue can have any positive or negative effect
• When to use: When you have one clear primary outcome

2. Any Rule: improvement on one of the outcomes

• Tests if at least one outcome improves: OR
• Consists of two decision regions
• Most liberal: accepts marginal improvements, even when the other outcome has a large decline
  
• When to use: Rare, unless any improvement is clinically meaningful, regardless of decline on the other outcome

3. All Rule: improvement on all of the outcomes

• Tests if both improve: AND
• Most conservative: requires improvement everywhere
• When to use: When improvement on all outcomes is truly necessary (safety + efficacy)

4. Compensatory Rule: improvement on a weighted combination of outcomes

• Tests if weighted sum exceeds zero:
• The diagonal dashed line shows the decision boundary and corresponds to weights of for each outcome (see below for other weight combinations)
• Allows tradeoffs: A strong gain in one outcome can compensate for a small loss in another
• Allows different importances: Outcomes are weighted by their importances
• When to use: When outcomes are both meaningful but have different importances; when a small decline can be compensated for by a larger improvement

In these plots, bivariate (posterior) distributions of treatment differences are shown. The darker green parts of the distributions show the proportions of draw that fall in the region where superiority is concluded. The light green parts of the distribution falls outside of the superiority region. If a sufficiently large part of the posterior distribution (e.g., 95%) falls in the superiority region, superiority is concluded. Note that for the Any rule a multiple testing correction on the cutoff is needed.

Superiority decisions with different rules

As can be seen in the plots above as well, each of these rules results in different amounts of evidence of superiority. The posterior probabilities to conclude superiority for the visualized posterior distributions are the following:

Posterior Probabilities of Superiority for Four Decision Rules:

                              Rule Posterior_Probability
1                  Single (δ₁ > 0)                 0.843
2           Any (δ₁ > 0 OR δ₂ > 0)                 0.992
3          All (δ₁ > 0 AND δ₂ > 0)                 0.666
4 Compensatory (0.5δ₁ + 0.5δ₂ > 0)                 0.950

From these posterior probabilities, it can be seen that the All rule is the most strict rule with the lowest posterior probability, while the Any rule is the most lenient rule. While this is generally the case, it is useful to note that the relation between decision rules and correlation is a bit more complicated. There will be a post on this topic later.

Weight specifications in the compensatory decision rule

As mentioned above, the compensatory rule can can deal with different importances.
Therefore we must decide how much does each outcome contributes to the decision (i.e., conclusion regarding superiority in this case). This is where weights come in.

A weighted compensatory decision rule uses the formula:

where:

• and are weights, typically summing to
• is the treatment effect on outcome 1
• is the treatment effect on outcome 2

The weights quantify how much each outcome contributes to the overall treatment success decision. A higher weight means that the accomponying outcome is more critical to declaring the treatment a success.

Below, we showcase five different weight specifications in more detail.

Posterior probabilities for different weight specifications

                    Specification Posterior_Probability
1   Fatigue priority (0.25, 0.75)                 0.908
2              Equal (0.50, 0.50)                 0.950
3 Moderate cognitive (0.60, 0.40)                 0.947
4 Cognitive priority (0.75, 0.25)                 0.920
5 Cognitive dominant (0.90, 0.10)                 0.873

From the output above, it can be seen that different weight specifications result in different posterior probabilities of superiority. Hence, weights are not technical details. They formalize what the decision-makers value most. Choosing weights thoughtfully is essential for multi-outcome decision-making.

# Install bmco if needed
# install.packages("bmco")
library(bmco)

# Create example binary outcome trial data
# In practice, this would come from your actual trial
set.seed(2024)
n_per_group <- 200

# Generate binary outcomes consistent with the posterior parameters
trial_data <- data.frame(
  treatment = c(rep("control", n_per_group), rep("new", n_per_group)),
  cognitive_function = c(
    rbinom(n_per_group, 1, 0.40),      # Control: 40% success
    rbinom(n_per_group, 1, 0.65)       # New: 65% success (IMPROVED)
  ),
  fatigue_reduction = c(
    rbinom(n_per_group, 1, 0.50),      # Control: 50% success
    rbinom(n_per_group, 1, 0.35)       # New: 35% success (WORSENED)
  )
)

Success rates by group:

 COGNITIVE FUNCTION:
   Control:      0.425 
   New treatment:  0.61 
   Difference:   0.185 

 FATIGUE REDUCTION:
   Control:      0.575 
   New treatment:  0.345 
   Difference:   -0.23 

result_single <- bmvb(
  data = trial_data,
  grp = "treatment",
  grp_a = "control",
  grp_b = "new",
  y_vars = c("cognitive_function", "fatigue_reduction"),
  rule = "Comp",
  w = c(1,0),  # Focus only on cognitive function
  n_it = 5000
)

p_single <- result_single$delta$pop[1]

 RESULT:
 P(Superiority) =  1 

result_all <- bmvb(
  data = trial_data,
  grp = "treatment",
  grp_a = "control",
  grp_b = "new",
  y_vars = c("cognitive_function", "fatigue_reduction"),
  rule = "All",
  n_it = 5000
)

p_all <- result_all$delta$pop[1]

RESULT:
 P(Superiority) =  0 

result_any <- bmvb(
  data = trial_data,
  grp = "treatment",
  grp_a = "control",
  grp_b = "new",
  y_vars = c("cognitive_function", "fatigue_reduction"),
  rule = "Any",
  n_it = 5000
)

p_any <- result_any$delta$pop[1]

 RESULT:
 P(Superiority) =  1 

# ===== Example 1: Fatigue Priority (0.25, 0.75) =====
result_fatigue_priority <- bmvb(
  data = trial_data,
  grp = "treatment",
  grp_a = "control",
  grp_b = "new",
  y_vars = c("cognitive_function", "fatigue_reduction"),
  rule = "Comp",
  w = c(0.25, 0.75),       # Cognitive 25%, fatigue 75%
  n_it = 5000
)

 δ₁ (cognitive)  =  0.185 
 δ₂ (fatigue)    =  -0.23 
 Weighted delta:
 0.25·δ₁ + 0.75·δ₂ = 0.25· 0.185  + 0.75· -0.23  =  -0.126 

 Fatigue priority (0.25, 0.75): P(superiority) = 0 

# ===== Example 2: Equal Weights (0.50, 0.50) =====
result_equal <- bmvb(
  data = trial_data,
  grp = "treatment",
  grp_a = "control",
  grp_b = "new",
  y_vars = c("cognitive_function", "fatigue_reduction"),
  rule = "Comp",           # Use Compensatory
  w = c(0.50, 0.50),       # Equal weights
  n_it = 5000
)

 Weighted delta:
 0.50·δ₁ + 0.50·δ₂ = 0.50· 0.185  + 0.50· -0.23  =  -0.022 

 Equal weights (0.50, 0.50): P(superiority) = 0.245 

# ===== Example 3: Moderate Cognitive Emphasis (0.60, 0.40) =====
result_moderate <- bmvb(
  data = trial_data,
  grp = "treatment",
  grp_a = "control",
  grp_b = "new",
  y_vars = c("cognitive_function", "fatigue_reduction"),
  rule = "Comp",
  w = c(0.60, 0.40),       # Cognitive 60%, fatigue 40%
  n_it = 5000
)

 Weighted delta:
 0.60·δ₁ + 0.40·δ₂ = 0.60· 0.185  + 0.40· -0.23  =  0.019 

 Moderate cognitive emphasis (0.60, 0.40): P(superiority) = 0.716 

# ===== Example 4: Cognitive Priority (0.75, 0.25) =====
result_cog_priority <- bmvb(
  data = trial_data,
  grp = "treatment",
  grp_a = "control",
  grp_b = "new",
  y_vars = c("cognitive_function", "fatigue_reduction"),
  rule = "Comp",
  w = c(0.75, 0.25),       # Cognitive 75%, fatigue 25%
  n_it = 5000
)

 Weighted delta:
 0.75·δ₁ + 0.25·δ₂ = 0.75· 0.185  + 0.25· -0.23  =  0.081 

 Cognitive priority (0.75, 0.25): P(superiority) = 0.982 

# ===== Example 5: Cognitive Dominant (0.90, 0.10) =====
result_cog_dominant <- bmvb(
  data = trial_data,
  grp = "treatment",
  grp_a = "control",
  grp_b = "new",
  y_vars = c("cognitive_function", "fatigue_reduction"),
  rule = "Comp",
  w = c(0.90, 0.10),       # Cognitive 90%, fatigue 10%
  n_it = 5000
)

 Weighted delta:
 0.90·δ₁ + 0.10·δ₂ = 0.90· 0.185  + 0.10· -0.23  =  0.144 

 Cognitive dominant (0.90, 0.10): P(superiority) = 0.998 

                                            Rule         Delta P_Superiority
1                                         Single 0.183, -0.227         1.000
2                                            All 0.183, -0.228         0.000
3                                            Any 0.183, -0.227         1.000
4   Compensatory - Fatigue priority (0.25, 0.75)        -0.126         0.000
5              Compensatory - Equal (0.50, 0.50)        -0.023         0.245
6 Compensatory - Moderate cognitive (0.60, 0.40)         0.019         0.716
7 Compensatory - Cognitive priority (0.75, 0.25)          0.08         0.982
8 Compensatory - Cognitive dominant (0.90, 0.10)         0.142         0.998

Takeaway

The decision rule you choose shapes your entire trial:

• Single: Use only for truly secondary outcomes
• Any: Too liberal for most clinical contexts
• All: Only if all outcomes are absolutely necessary
• Compensatory: The new default choice; lets you encode clinical judgment about tradeoffs

The beauty of a multivariate framework with a weighted combination is that you can make your decision rule match your actual clinical question.

Further reading:

• Murray, T., Thall, P. & Yuan, Y. (2016). Utility-based designs for randomized comparative trials with categorical outcomes. Statistics in medicine.
• Kavelaars, X., Mulder, J. & Kaptein, M. (2020). Decision-making with multiple correlated binary outcomes in clinical trials. Statistical Methods in Medical Research. https://doi.org/10.1177/0962280220922256
• Sozu, T., Sugimoto, T. & Hamasaki, T. (2010). Sample size determination in clinical trials with multiple co-primary binary endpoints.. Statistics in medicine. https://doi.org/10.1002/sim.3972
• Sozu, T., Sugimoto, T. & Hamasaki, T. (2016). Reducing unnecessary measurements in clinical trials with multiple primary endpoints. Journal of biopharmaceutical statistics.
• Su, T., Glimm, E., Whitehead, J. & Branson, M. (2012). An evaluation of methods for testing hypotheses relating to two endpoints in a single clinical trial. Pharmaceutical statistics.

Questions? Any references that should be included as well? Found this useful? I’m on social media and happy to discuss!

The challenge with standard approaches

Testing outcomes separately (symptom relief indpendently from QoL) invites complications:

• Often outcomes are given a primary or secondary role in analysis and decision-making. But that is at least problematic when safety and effectivity are measured together. Safety is almost never of secondary concern, and neither is effectivity.
• Decisions are often based on an ad hoc basis: Two decisions are combined into a single decision, without providing a probability statement on the combined decision, leaving the accuracy of the final decision less transparent.
• Your two outcomes may be correlated: if the treatment relieves symptoms, it might as well improve quality of life too. Standard separate testing doesn’t easily account for this structure.

These methods do not always answer the most relevant question. The real question people care about:

“What is the probability that this treatment improves BOTH outcomes?”

Or: “At least one of the outcomes?” Or: “This weighted combination of outcomes?”

These are natural, practical questions, but they are not straightforward to answer with standard approaches - especially when outcomes differ in their relative contributions to success.

A direct approach: Joint modeling of binary outcomes

Here’s an elegant solution: Model both outcomes together in a single framework, account for their natural correlation, and use the posterior distribution to answer your specific question.

The bmco package in R implements this. It’s built on the principle that when you have multiple binary outcomes, analyzing them jointly — rather than piecing together separate tests - gives you more coherence.

Let’s simulate trial data:

library(bmco)
set.seed(2026)

# Simulate a trial: 100 patients, 50 per arm

outcome_data_placebo <- t(rmultinom(50, 1, c(0.30, 0.10, 0.20, 0.20)))  
# Placebo: 40% pain relief, 50% mobility improvement, correlation 0.33

outcome_data_drug <- t(rmultinom(50, 1, c(0.55, 0.10, 0.20, 0.15)))   
# Drug: 65% pain relief, 70% mobility improvement, correlation 0.30

trial_data <- data.frame(
  treatment = rep(c("placebo", "drug"), each = 50),
  pain_relief = c(
  rowSums(outcome_data_placebo[,c(1,2)]), 
  rowSums(outcome_data_drug[,c(1,2)])
  ),
  mobility_improved = c(
     rowSums(outcome_data_placebo[,c(1,3)]), 
     rowSums(outcome_data_drug[,c(1,3)])
  )
)

head(trial_data)

  treatment pain_relief mobility_improved
1   placebo           1                 1
2   placebo           0                 0
3   placebo           0                 0
4   placebo           1                 1
5   placebo           0                 0
6   placebo           1                 1

Now, here’s the key question for this trial: What is the probability that the drug is better on a relevant combination of pain relief and mobility improvement?

With bmco, you get a direct answer:

result <- bmvb(
  data = trial_data,
  grp = "treatment",
  grp_a = "placebo",
  grp_b = "drug",
  y_vars = c("pain_relief", "mobility_improved"),
  rule = "All",  # Drug must improve BOTH outcomes
  test = "right_sided",
  n_it = 5000
)

print(result)

Multivariate Bernoulli Analysis
=================================

   Group mean y1 mean y2
 placebo   0.442   0.443
    drug   0.596   0.865
  n(placebo) = 50    n(drug) = 50

Posterior probability P(drug > placebo) [All rule]: 0.940

Use summary() for credible intervals and ESS.

Output for decision-making:

Posterior probability P(drug > placebo on BOTH outcomes): 0.94

If the standard superiority criterion of posterior probability > 0.95 (one-sided testing) is used, you would conclude that the treatment is better on both outcomes.

Why is this nice? The analysis relies upon a multivariate distribution and naturally respects the correlation between your outcomes. If pain relief and mobility are related, the model captures that. And you get a probability statement that can be used for superiority and inferiority decision with control of false positives (i.e., Type I error). The latter is convenient: When an uninformative prior is used, the posterior probability has a direct relationship to the p-value.

Three decision rules compared

Same data, different success criteria:

result_all <- bmvb(
  data = trial_data,
  grp = "treatment",
  grp_a = "placebo",
  grp_b = "drug",
  y_vars = c("pain_relief", "mobility_improved"),
  rule = "All",
  n_it = 5000,
  return_samples = TRUE # For plot of sample
)

cat("P(drug > placebo on BOTH):", result_all$delta$pop)

P(drug > placebo on BOTH): 0.946

result_any <- bmvb(
  data = trial_data,
  grp = "treatment",
  grp_a = "placebo",
  grp_b = "drug",
  y_vars = c("pain_relief", "mobility_improved"),
  rule = "Any",
  n_it = 5000,
  return_samples = TRUE
)

cat("P(drug > placebo on AT LEAST ONE):", result_any$delta$pop)

P(drug > placebo on AT LEAST ONE): 1

result_comp <- bmvb(
  data = trial_data,
  grp = "treatment",
  grp_a = "placebo",
  grp_b = "drug",
  y_vars = c("pain_relief", "mobility_improved"),
  rule = "Comp",
  w = c(0.70, 0.30),  # Pain relief is 70% of the story, mobility 30%
  n_it = 5000,
  return_samples = TRUE
)

cat("P(weighted improvement):", result_comp$delta$pop)

P(weighted improvement): 0.999

Beyond the simple case: subgroups and clustering

Additional advantage comes when your data are more complex:

# Add age data 
trial_data_with_age <- trial_data
trial_data_with_age$age <- round(rnorm(100, 60, 8))

# Does the drug improve BOTH outcomes in elderly patients (age 65+)?
result_elderly <- bglm(
  data = trial_data_with_age,
  grp = "treatment",
  grp_a = "placebo",
  grp_b = "drug",
  x_var = "age",
  y_vars = c("pain_relief", "mobility_improved"),
  x_method = "Empirical",
  x_def = c(65, Inf),  # Elderly patients
  rule = "All",
  n_it = 5000
)

cat("P(drug > placebo on BOTH | age 65+):", result_elderly$delta$pop)

P(drug > placebo on BOTH | age 65+): 0.635

# Use in-built multilevel data 
trial_data_with_hospitals <- bglmm_data
colnames(trial_data_with_hospitals) <- c("hospital", "treatment", "age", "pain_relief", "mobility_improved")
head(trial_data_with_hospitals)

  hospital treatment      age pain_relief mobility_improved
1        1   placebo 41.27538           0                 1
2        1   placebo 49.95309           1                 1
3        1   placebo 54.95375           1                 1
4        1   placebo 69.72819           1                 1
5        1   placebo 38.03138           1                 0
6        1   placebo 51.30438           1                 0

result_clustered <- bglmm(
  data = trial_data_with_hospitals,
  grp = "treatment", 
  grp_a = "placebo",
  grp_b = "drug",
  id_var = "hospital",  # Hospital as a random effect
  x_var = "age",
  y_vars = c("pain_relief", "mobility_improved"),
  rule = "All",
  n_burn = 10000,
  n_it = 20000 # Takes some time; large number of iterations needed
)

cat("P(drug > placebo on BOTH | accounting for hospitals):", result_clustered$delta$pop)

P(drug > placebo on BOTH | accounting for hospitals): 0.969

When this approach is useful

In applied research, several things are typically true:

1. You care about multiple outcomes. The treatment affects different aspects of patient well-being, educational progress, quality of life — or any outcome your domain measures.

2. Those outcomes are often correlated. A good treatment usually helps on multiple fronts; a poor one hurts on multiple fronts.

3. You want a clear decision rule. Does this treatment meet our success criteria? And how confident are we?

The joint modeling approach handles all three naturally:

• Flexible decision rules: All (conjunctive), Any (disjunctive), or Compensatory (weighted): Three different ways to capture relative importance of outcomes and to define “success” according to your contextual needs.
• Efficient use of data: The correlation structure gets learned and used. You don’t lose information to assuming independence and may sometimes even end up with a lower sample size.
• Interpretable results: Posterior probabilities statements produce coherent decisions with Type I error control.

Getting Started

Install from CRAN:

install.packages("bmco")

Then load the package and explore the vignettes. Here, more details on MCMC settings, prior choices, and other options are given.

library(bmco)
vignette("Introduction to bmco")
vignette("Subgroup Analysis with Multivariate Binary Outcomes")
vignette("Subgroup Analysis with Multivariate Binary Outcomes in Multilevel Data")

Each vignette walks through realistic examples with full reproducible code.

Power analysis is available in the bmco-pwr Shiny app for basic multivariate Bernoulli analysis using bmvb() and subgroup analysis via bglm().

Quick Reference

References

The methods are published in peer-reviewed journals:

• Basic multivariate Bernoulli: Kavelaars, X., Mulder, J. & Kaptein, M. (2020). Decision-making with multiple correlated binary outcomes in clinical trials. Statistical Methods in Medical Research. https://doi.org/10.1177/0962280220922256
• Subgroup analysis: Kavelaars, X., Mulder, J. & Kaptein, M. (2024). Bayesian Multivariate Logistic Regression for Superiority and Inferiority Decision-Making under Observable Treatment Heterogeneity. Multivariate Behavioral Research. https://doi.org/10.1080/00273171.2024.2337340
• Multilevel models: Kavelaars, X., Mulder, J. & Kaptein, M. (2023). Bayesian multilevel multivariate logistic regression for superiority decision-making under observable treatment heterogeneity. BMC Medical Research Methodology. https://doi.org/10.1186/s12874-023-02034-z

Questions? Found this useful? I’m on social media and happy to discuss!