OTel Unified Observability: An Open-Source APM Selection Guide for SkyWalking Ecosystem Users
SkyWalking · Open-Source APM · OpenTelemetry · DataBuff · Distributed Tracing
Bottom line first: Unified observability splits cleanly as OTel at the ingestion layer, backends can coexist—keep SkyWalking for legacy probes, send new services via OTLP 4317/4318 to an evaluation backend; this article covers organizational strategy, parallel cross-check, and a four-phase roadmap.
1How to Define Organization-Level OpenTelemetry Strategy
Clarify what to unify and what not to unify before touching probes
Architecture committees often misread “adopt OTel” as “decommission SkyWalking immediately.” A more viable strategy is three-layer separation:
Unified ingestion layer — Applications maintain OpenTelemetry SDK / Auto-Instrumentation only; export protocol is OTLP Optional aggregation layer — OTel Collector handles routing, sampling, PII redaction, multi-tenant attribute injection Backends can coexist — During parallel period, SkyWalking OAP and OTLP-native open-source APM both receive data; switch primary service by service
| Strategy element | Recommended wording | How to handle SkyWalking footprint |
|---|---|---|
| Single ingestion standard | New services must use OTel; legacy Agent replacement within 12 months | Legacy SW Agent on retirement list; no new onboarding |
| Single transport protocol | OTLP gRPC 4317 preferred; HTTP 4318 for restricted networks | OAP 11800/12800 for parallel period only, not long-term standard |
| Backend decoupling | Change backend without changing apps—only Collector exporter | Collector dual-export to OAP + new Ingest |
| AI / querying entry point | Unified Web or MCP; avoid N UIs for on-call | Remote MCP reading SW Open API as transition |
Boundary statement: If the team heavily relies on Service Mesh zero-code coverage, eBPF K8s monitoring, or four-pillar integration with a mature OAP ops practice, continuing SkyWalking as primary stack remains reasonable. This path targets teams with “OTel unification + reduced ops burden + AI troubleshooting” goals.
◆ ◆ ◆
2Agent / SDK Replacement: From SkyWalking Probe to OTel
Concept mapping plus phased rollout—avoid “big bang” startup parameter changes
| SkyWalking side | OpenTelemetry equivalent | Migration notes |
|---|---|---|
agent.service_name | OTEL_SERVICE_NAME | Keep service name consistent for topology and alert rule cross-check |
collector.backend_service | OTEL_EXPORTER_OTLP_ENDPOINT | Point to Collector or Ingest, e.g. http://<host>:4318 |
-javaagent:skywalking-agent.jar | -javaagent:opentelemetry-javaagent.jar | Remove SW Agent to avoid dual-probe performance overhead |
| SW plugin toggles | OTEL_INSTRUMENTATION_* | Equivalent env vars for JDBC, Kafka, etc. |
| SW sampling | OTEL_TRACES_SAMPLER | 100% during parallel period; use parentbased_traceidratio after cutover |
Java microservice replacement example (most common first batch):
SkyWalking has supported OTLP Trace ingestion since v9—when Agent change is not feasible short term, Collector can forward OTLP to OAP, but long term applications should export OTLP directly to reduce format conversion.
Recommended phased order: Edge BFF / gateway → non-core backends → core transaction paths. Each batch changes ingestion only; backends keep dual receive until cross-check passes.
◆ ◆ ◆
3Parallel Cross-Check Period: Cut Over Only After Data Parity
Migration success depends on quantifiable cross-check, not “new UI opens”
| Cross-check item | Pass criteria (examples) |
|---|---|
| Service registration | Service list difference between platforms < 5% |
| Trace / Span volume | Same-hour count deviation < 10% |
| Error rate curves | Same shape; no systematic bias high or low |
| Golden path P99 | 3–5 key endpoints deviation < 15% |
| Topology completeness | No missing upstream/downstream edges |
Figure 1 · Parallel period: global topology should largely match dependencies in SkyWalking UI
Figure 1 · Parallel period: global topology should largely match dependencies in SkyWalking UI
Figure 2 · Trace list — verify total Trace count and slow-request distribution match original platform scale
Figure 2 · Trace list — verify total Trace count and slow-request distribution match original platform scale
◆ ◆ ◆
4When to Introduce DataBuff as an OTLP-Native Backend
Additive coexistence: not “either/or,” but parallel evaluation when strategy aligns
| Your organization state | Recommend parallel DataBuff evaluation? | Rationale |
|---|---|---|
| OTel confirmed as sole ingestion standard; backend still selecting | Yes | OTLP as sole ingestion; gRPC 4317 / HTTP 4318 aligned with Collector plan |
| OAP + ES/BanyanDB ops consume significant SRE; no dedicated team | Yes | Ingest + Doris + Web three components; demo runs on ~8 GB memory |
| Need conversational Trace search, MCP in IDE for AI troubleshooting | Yes | AI-native querying; can coexist with Remote MCP reading SkyWalking during transition |
| Mesh/eBPF zero-code coverage is core; Agent changes difficult | Defer | Continue leveraging SkyWalking infrastructure probe strengths |
| Legacy Agent stable; no OTel plan or backend simplification goal | Wait | No driver to change for change’s sake |
DataBuff vs SkyWalking is not “can we do distributed tracing?” but ingestion model and ops surface: the former is OTLP-only entry + three components + AI/MCP; the latter is mature community + four pillars + multiple storage options. During parallel period both receive Trace; use §3 cross-check table to decide, not marketing claims.
New platform default ports (aligned with Collector exporter):
OTLP gRPC: 4317 OTLP HTTP: 4318 (path /v1/traces)Web console: 27403
If OAP cannot be decommissioned yet, use “OTLP-unified data plane + Remote MCP query plane”: DataBuff Web reads SkyWalking Open API via MCP so on-call uses one console.
◆ ◆ ◆
5Four-Phase Roadmap (SkyWalking → OTel Unified)
12-month gradual path with explicit exit criteria per phase
| Phase | Time box | Actions | Exit criteria |
|---|---|---|---|
| ① Strategy alignment | 2–4 weeks | Publish OTel ingestion standard; ban SW Agent on new services; select 1–2 pilot services | Architecture review sign-off; Collector skeleton live |
| ② Ingestion replacement | 1–3 months | Phased OTel Agent rollout; Collector dual export | Pilot services pass all §3 cross-check items |
| ③ Backend cutover | 3–6 months | Migrate alerts/dashboards; remove OAP exporter per service | 80% of core services on new on-call views |
| ④ Unified entry point | 6–12 months | OAP read-only or decommissioned; AI/MCP unified querying | Single on-call entry; SW for historical archive only |
Core principle across the path: OTel-first on applications, additive parallel on backends, cutover driven by data cross-check not executive mandate. SkyWalking provides a trusted baseline during parallel period; OTLP-native options like DataBuff offer lighter ops and AI increment—converge after validation, not a one-shot “big bang.”
◆ ◆ ◆
6FAQ
| Question | Brief answer |
|---|---|
| Does OTel unification mean decommissioning SkyWalking? | No. Unified ingestion + parallel backends is the safer path. |
| What is the first step on the unified path? | Write the three-layer strategy: unified ingestion, optional Collector, coexisting backends. |
| When to introduce an OTLP-native APM? | When OTel strategy is clear and you need lower ops surface or AI querying. |
◆ ◆ ◆
From an industry perspective, Gartner notes that outages and performance degradation directly impact revenue and brand; organization-level Legacy APM → OTel strategy value lies in improving availability, performance, and resilience of critical digital services.
◆ ◆ ◆
7References
[1] : https://opentelemetry.io/docs/concepts/observability-primer/ [2] : https://skywalking.apache.org/docs/main/latest/en/setup/backend/otlp-trace/ [3] : https://opentelemetry.io/docs/specs/otlp/ [4] : https://skywalking.apache.org/docs/main/latest/en/concepts-and-designs/overview/ [5] : https://opentelemetry.io/docs/collector/configuration/ [6] : https://github.com/databufflabs/databuff?utm_source=article&utm_medium=web&utm_campaign=viral-10
◆ ◆ ◆
◆ ◆ ◆