Pull drm updates from Dave Airlie:
 "Highlights:

   - core: a lot of atomic refactoring in progress, panic got split into
     a core/helpers model

   - amdgpu: enables HDMI 2.1 FRL and other good things finally, UAlink
     over Ethernet support

   - xe: has a bunch of CRI support, ras error for bad vram, memory
     offlining support.

   - msm: enables a bunch more GPUs.

  Otherwise it's a the usual mountain of stuff all over the place.
  Details:

  rust:
   - I/O tree shared with driver-core
   - DMA tree shared with driver-core
   - allow aux registration data closures to return covariant sub-fields
   - Implement Send and Sync for MapleTree
   - add pci resource flags and IORESOURCE_MEM_64

  drm/rust:
   - add dma_fence abstraction
   - init DRM registrations and their data in place
   - drop unused flags argument to registration

  vgaarb:
   - rework API to pass a private ptr instead of pci device

  sound:
   - adapt to omapdrm changes

  gem:
   - add callback for handle count going to 0

  gpusvm:
   - share one HMM fault and keep single mappings inline

  pagemap:
   - Prevent CPU stalls during unbounded pagemap teardown

  dma-fence:
   - fix docs

  vga_switcheroo:
   - disconnect from fbcon
   - update fbcon from within fbdev emulation

  atomic:
   - move DRM from state reset callbacks to atomic_create_state
   - remove drm_simple_encoder_init

  panic:
   - split into core and helpers

  buddy:
   - improve dirty tracking

  clients:
   - improve vmap handling
   - fbdev: Move helpers from drm_kms_helper.ko to drm_client_lib.ko

  display:
   - export HDMI SCDC status data via debugfs
   - rework SCDC for better HDMI 2.0 support
   - dp/mst: improve DPCD compliance

  edid:
   - parse AMD VSDB
   - parse ALLM/VRR capabilities
   - support detecting panel size

  log:
   - add drm_warn_ratelimited

  sched:
   - cleanup interface
   - add missing locking
   - fix run-queue cleanup
   - fix locking

  sysfb:
   - remove obsolete quirk for Lenovo D330
   - fix panel orientation for non-native display modes
   - drop drm_class_device_register

  colorop:
   - provide DRM_COLOR_OP_FIXED_MATRIX

  bridge:
   - analogix: Support eDP lane mapping
   - ti-sn65dsi83: Support LVDS Channel B on SN65DSI84
   - Select DRM_PANEL_BRIDGE for various drivers that require it
   - add chrontel ch7218a support

  panel:
   - panel-edp: Support MNE007QS3-F, TM140VDXP15, and KD116N36-30NB-A001
   - samsung-s6d16d0: Use mipi_dsi_*_multi() functions
   - support Ilitek ILI7836A OLED plus DT bindings
   - support Novatek NT36532 plus DT bindings
   - convert several drivers to managed cleanup
   - fix Kconfig selections
   - edp:
      - Add support for BOE NV140FHM-T0A
      - Support CSOT MNE007QB1-1
      - NV140WU2-M70
   - ebbg-ft8719: Fix MODULE_LICENSE() string; Improve initialization
   - samsung: Share test-key helpers
   - ilitek-ili9882t: Support Ilitek IL79600A plus DT bindings
   - simple: Support Truly TFT320240-229-E plus DT bindings; Support BOE
     DV215FHM-R01 plus DT bindings
   - Refactor struct drm_panel to contain struct drm_bridge plus fallout
     in various panel drivers
   - Add support for BOE Novatek NT51021, Retroid Pocket 6, Raydium
     RM69220 DDIC panels, Sharp LQ138P1JX61 eDP, BOE NV140FHD-N43, AUO
     B140HAN04.0, AUO 0x0db3, panels.

  amdgpu:
   - use AMD VSDB for freesync
   - implement YUV-to-RGB with fixed-matrix colorop
   - enable FRL for HDMI
   - HDMI 2.1 freesync support
   - HDMI 2.1 ALLM support
   - HDMI 2.1 VRR support
   - UALoE (UALink over Ethernet) support
   - add syncobj_points to userq signal ioctl
   - drop unused kunit tests
   - NBIO/GC/MMHUB/SMU updates/fixes
   - GPU spatial partitioning updates
   - display kunit fixes
   - userq fixes
   - RGB quantization fixes
   - DC RMCM refactor
   - DP MST fixes
   - USB4 fixes
   - add new trace events
   - Rework display instance handling
   - stack frame size fixes for various compilers/configs
   - VCN 4/5 reset fixes
   - freesync updates
   - cursor refactor
   - Fix direct scanout alpha on some DRM_FORMATs
   - Expand Mac reserved memory workaround
   - Make runtime pm more fine grained

  amdkfd:
   - updates and fixes
   - GC 10 CU occupancy support

  radeon:
   - eDP fix for DCE3.2 iMacs

  xe:
   - add drm_ras netlink error event support
   - add memory page offlining support
   - introduce error threshold to drm_ras
   - fix stale GTT mappings on unmap
   - reject page faults for non-page fault mode scratch VM
   - enable CRI package and VRAM temperatures
   - introduce cold reset recovery
   - handle CRI+ power interval encoding
   - VF migration for multi-queue
   - handle exec queue teardown after hot unplug
   - alerts and controller enabling modifications
   - use IBCv3 on PTL
   - VRAM health check
   - report GT TDR and VM rebind faults using SIGID
   - add multi_queue_active_lrca debugfs
   - shrinker: Runtime PM reference for non-system memory
   - guc: Add simple KLV decoder helpers
   - debugfs: Expose per-engine idle status
   - Keep walking on SVM eviction failure
   - Nuke PTs only after unlinking contested VMAs
   - limit sg segment size to PAGE_SIZE on Xen PV
   - keep VF LMEM bar size low if no VFs
   - CPU binds and ULLS on migration queue
   - New PTL PCI ID
   - Enable PXP for NVL-S
   - Do not dma_fence_put zero fences in err_out path
   - Refresh TTM LRU on SVM prefetch
   - export device UID in sysfs

  i915:
   - rename GEN6_PCODE_DATA to match spec
   - fix incorrect RCU teardown order
   - fix potential UAF in command parsing
   - avoid WARN splat during RPS boosting race
   - use simpler min functions
   - Fixes to make i915 work well with PREEMPT_RT

  i915/display:
   - Enable CMRR (Content Match Refresh Rate) in fixed-RR VRR path
   - Switch DG2 MPLL to DPLL framework
   - DP-HDMI PCON DSC fixes
   - limit invalid joiner combinations
   - CDCLK related fixes and refactor around PCODE integration
   - Gate UHBR SST SDP splitting on sink caps
   - add HDMI TMDS audio bandwidth check
   - vswing/preemph override
   - enable AS SDP Skip frames
   - Handle negative configuration index on DP link caps for robustness
   - Preparation for enabling SDR plane color pipeline
   - An eDP quirk for HP Pavilion Plus
   - Introduce PME (Power Management Event) support on display hotplug
   - Handle VSC SDP revision 7 in unpack

  msm:
   - CI:
      - Dropped sc7180 trogdor lazor from the CI
   - Core:
      - Fixed msm_ioremap for resources without a name
      - Changed msm.separate_gpu_kms to default to auto selection
   - GPU:
      - Enable CLX (Current Limit eXtension) for gen8 to avoid exceeding
        limits on CX/GX rails
      - New GPUs: a830 (Pakala), a845 (Maili), a850 (Hawi), x2-85
        (Mahua)
      - Fix VBIF halt mask on a306, which was preventing runpm suspend
      - Enable THP support for GEM buffers
      - Add modparam to disable waiting on fences in shrinker
      - Drop use of drm_prime_sg_to_page_array()
      - a5xx: don't read the GPMU registers on GPUs without a GPMU
      - a2xx/a3xx/a5xx: hangcheck progress detection
      - a6xx: gmuwrapper runpm fix
      - Calculate reported GMEM size based on active slices
      - Split a6xx/a7xx/a8xx catalog
   - DP:
      - Picked up some preparation patches for the MST support
      - Fixed reported status of the link training
   - DPU:
      - Fixed CRTC bandwidth computation
      - Fixed DSC programming
      - Fixed encoder disablement path
   - DSI:
      - Fixed VM leak in error path
      - Indention fix
   - MDP5:
      - Added MSM8952 configuration
      - Dropped incorrect cursor pipe on MSM8976/56
   - HDMI:
      - Small fixes

  nouveau:
   - improve runtime PM on R570 GSP firmware
   - various fixes throughout the driver
   - dispnv50: support 2.147 GHz pixel clock in GB20x

  nova:
   - add initial uapi concept
   - add initial PRAMIN access support
   - add GpuMm centralized memory manager
   - add GPU VM and BAR1 support
   - add GPU interrupts and GSP events support
   - GSP messaging fixes

  tyr:
   - use bitfield!
   - zero unsupported and newly added GPU info fields
   - limit doorbell array to 17 entries

  exynos:
   - FIMD fixes
      - Fix clock leak on resume error path
      - Fix Sparse warnings by adding missing __iomem annotation
   - VIDI cleanup
      - Replace kmalloc() and copy_from_user() with memdup_user()

  amdxdna:
   - fix BO handling
   - Support AIE2 rev 9
   - Fix UAF access in scheduler fences
   - Update workqueue interface
   - expose busy time via fdinfo

  ast:
   - support 256-byte EDID

  hyperv_drm:
   - remove support for pre win10 hosts

  komeda:
   - fix GLB_CORE_ID usage

  omap:
   - report HDMI hotplug events to ASoC HDMI codec

  panthor:
   - provide gpu_cache_flush tracepoint
   - improve dma_fence signalling latency
   - improve locking
   - export scheduling information via debugfs
   - Validate user-space input against firmware limits

  qaic:
   - reject BOs that exceed maximum page count
   - add missing include statements

  verisilicon:
   - fix hardware cursor offsets

  vkms:
   - implement YUV-to-RGB with fixed-matrix colorop

  gma500:
   - avoid unsupported ARGB formats in primary plane

  hibmc:
   - handle vblank interrupts correctly

  ivpu:
   - add DRM_IVPU_CAP_CMDQ_SET_PRIORITY
   - support command-queue priority ioctl
   - Simplify NULL-pointer handling in cleanup code

  arcpgu:
   - replace simple-display pipeline with regular atomic helpers

  aspeed:
   - replace simple-display pipeline with regular atomic helpers

  gm12u320:
   - replace simple-display pipeline with regular atomic helpers

  imagination:
   - various improvement to firmware interaction
   - add job to pvr_job_submit_fw trace event

  mali-dp:
   - fix runtime PM leak

  mcde:
   - replace simple-display pipeline with regular atomic helpers

  p111:
   - replace simple-display pipeline with regular atomic helpers
   - drop unsupported alpha formats from primary plane

  rcar-du:
   - support Renesas R-Car V4H DSC
   - dsi: Support larger transfers with AXI memory
   - Support RZ/G3L SoC

  repaper:
   - replace simple-display pipeline with regular atomic helpers

  rockchip:
   - dsi: improve PHY timings

  ssd130x:
   - support Solomon SSD1351 OLED plus DT bindings
   - ssd133x: upgrade color format to RGB565
   - ssd132x: Set correct hardware window when clearing screen

  tve200:
   - replace simple-display pipeline with regular atomic helpers

  xen:
   - replace simple-display pipeline with regular atomic helpers

  v3d:
   - improve job handling under perfmon

  vc4:
   - reliably reset V3D
   - drop brcm,vc4-v3d compatible

  virtio:
   - fix damage handling
   - Support ARGB8888 for virtio primary planes.

  host1x:
   - Support partitioning

  imx:
   - Replace simple-display with regular atomic helpers

  mxsfb:
   - Support bus formats plus DT bindings

  tegra:
   - improve syncobj handling on submit

  panfrost:
   - add Mali-G68 support
   - improve GPU reset handling

  rockchip:
   - add HDMI 2.0 support to dw-hdmi-qp
   - improve HPD handling"

* tag 'drm-next-2026-10-21' of https://gitlab.freedesktop.org/drm/kernel: (2324 commits)
  drm/adp: Switch to drm_atomic_helper_connector_create_state
  drm/malidp: Convert to atomic_create_state
  drm/vc4: Switch to drm_atomic_helper_connector_create_state
  drm/sysfb: Switch to drm_atomic_helper_connector_create_state
  drm/solomon: ssd130x: Switch to drm_atomic_helper_connector_create_state
  drm/imx: ipuv3: Switch to drm_atomic_helper_connector_create_state
  drm/hyperv: Switch to drm_atomic_helper_connector_create_state
  drm/bridge: megachips: Switch to drm_atomic_helper_connector_create_state
  drm/bridge: lontium-lt8912b: Switch to drm_atomic_helper_connector_create_state
  drm/arm: komeda: Switch to drm_atomic_helper_connector_create_state
  drm/appletbdrm: Fix incorrect error message on encoder attach failure
  drm/sun4i: hdmi: Convert to drmm_connector_hdmi_init()
  drm/rockchip: rk3066_hdmi: Advertise HDMI 1.4 capabilities
  drm/bridge: lontium-lt9611: Advertise HDMI 1.4 capabilities
  drm/bridge: ite-it66121: Advertise HDMI 1.2 capabilities
  drm/bridge: ite-it6263: Advertise HDMI 1.3 capabilities
  drm/bridge: ite-it6263: Drop redundant .mode_valid hook
  drm/bridge: inno-hdmi: Advertise HDMI 1.2 capabilities
  drm/bridge: adv7511: Advertise HDMI 1.2 capabilities
  drm/sched: Document entity->stopped is protected by entity->lock
  ...
This commit is contained in:
Linus Torvalds
2026-10-23 18:29:02 -10:00
1860 changed files with 289923 additions and 34838 deletions
@@ -0,0 +1,10 @@
What: /sys/bus/pci/drivers/xe/.../device_uid
Date: October 2026
KernelVersion: 7.4
Contact: [email protected]
Description:
RO. Unique 64-bit identifier of the GPU SoC device, exposed as
hexadecimal value.
This sysfs file is present only on supported Intel Xe platforms.
Accessible only to users with administrative privileges.
+1
View File
@@ -289,6 +289,7 @@ Example of the output showing the implemented key value pairs:
drm-driver: amdxdna_accel_driver
drm-client-id: 3219
drm-pdev: 0000:c5:00.1
drm-engine-npu: 14003870 ns
amdxdna_accel_driver-heap-alloc: 60 KiB
amdxdna_accel_driver-internal-alloc: 67588 KiB
amdxdna_accel_driver-external-alloc: 0
@@ -119,11 +119,11 @@ examples:
#size-cells = <0>;
panel@0 {
compatible = "bananapi,lhr050h41", "ilitek,ili9881c";
reg = <0>;
power-supply = <&reg_display>;
reset-gpios = <&r_pio 0 5 1>; /* PL05 */
backlight = <&pwm_bl>;
compatible = "bananapi,lhr050h41", "ilitek,ili9881c";
reg = <0>;
power-supply = <&reg_display>;
reset-gpios = <&r_pio 0 5 1>; /* PL05 */
backlight = <&pwm_bl>;
};
port {
@@ -27,6 +27,9 @@ properties:
power-domains:
maxItems: 1
resets:
maxItems: 1
required:
- compatible
- reg
@@ -42,13 +42,22 @@ properties:
properties:
port@0:
$ref: /schemas/graph.yaml#/properties/port
description:
Input node to receive pixel data.
description: Input node to receive pixel data.
port@1:
$ref: /schemas/graph.yaml#/properties/port
description:
Port node with one endpoint connected to a dp-connector node.
$ref: /schemas/graph.yaml#/$defs/port-base
unevaluatedProperties: false
description: Port node with one endpoint connected to sink device node.
properties:
endpoint:
$ref: /schemas/media/video-interfaces.yaml#
unevaluatedProperties: false
properties:
data-lanes:
minItems: 1
maxItems: 4
items:
enum: [ 0, 1, 2, 3 ]
required:
- port@0
@@ -84,8 +84,8 @@ examples:
};
mipi_lvds_0_pxl2dpi_dc_pixel_link1: endpoint@1 {
reg = <1>;
remote-endpoint = <&dc_pixel_link1_mipi_lvds_0_pxl2dpi>;
reg = <1>;
remote-endpoint = <&dc_pixel_link1_mipi_lvds_0_pxl2dpi>;
};
};
@@ -107,11 +107,11 @@ examples:
};
port@1 {
reg = <1>;
reg = <1>;
endpoint {
remote-endpoint = <&hdmi_in>;
};
endpoint {
remote-endpoint = <&hdmi_in>;
};
};
};
};
@@ -104,7 +104,7 @@ examples:
endpoint {
remote-endpoint = <&stdp2690_in>;
};
};
};
};
};
};
@@ -106,7 +106,7 @@ examples:
video-ports = <0x230145>;
#sound-dai-cells = <1>;
/* DAI-format / AP_ENA reg value */
/* DAI-format / AP_ENA reg value */
audio-ports = <TDA998x_SPDIF 0x04>,
<TDA998x_I2S 0x03>;
@@ -0,0 +1,96 @@
# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
%YAML 1.2
---
$id: http://devicetree.org/schemas/display/bridge/renesas,r8a779g0-dsc.yaml#
$schema: http://devicetree.org/meta-schemas/core.yaml#
title: Renesas R-Car DSC Display Stream Compression
maintainers:
- Marek Vasut <[email protected]>
description: |
This binding describes the VESA DSC Display Stream Compression encoder
embedded in the Renesas R-Car V4H SoC. The encoder supports all DSC1.1
encoding mechanisms, configurable bits-per-pixel, resolution up to 8k.
properties:
compatible:
const: renesas,r8a779g0-dsc
reg:
maxItems: 1
clocks:
maxItems: 1
interrupts:
maxItems: 1
power-domains:
maxItems: 1
resets:
maxItems: 1
ports:
$ref: /schemas/graph.yaml#/properties/ports
properties:
port@0:
$ref: /schemas/graph.yaml#/properties/port
description: R-Car DU input port
port@1:
$ref: /schemas/graph.yaml#/properties/port
description: R-Car DSI output port
required:
- port@0
- port@1
required:
- compatible
- reg
- clocks
- interrupts
- power-domains
- resets
- ports
unevaluatedProperties: false
examples:
- |
#include <dt-bindings/clock/r8a779g0-cpg-mssr.h>
#include <dt-bindings/interrupt-controller/arm-gic.h>
#include <dt-bindings/power/r8a779g0-sysc.h>
bridge@feb8d000 {
compatible = "renesas,r8a779g0-dsc";
reg = <0xfeb8d000 0x400>;
interrupts = <GIC_SPI 559 IRQ_TYPE_LEVEL_HIGH>;
clocks = <&cpg CPG_MOD 2819>;
power-domains = <&sysc R8A779G0_PD_ALWAYS_ON>;
resets = <&cpg 2819>;
ports {
#address-cells = <1>;
#size-cells = <0>;
port@0 {
reg = <0>;
dsc_in_dsi1: endpoint {
remote-endpoint = <&du_out_dsi1>;
};
};
port@1 {
reg = <1>;
dsc_out_dsi1: endpoint {
remote-endpoint = <&dsi1_in>;
};
};
};
};
...
@@ -29,6 +29,7 @@ properties:
- adi,adv7123
- algoltek,ag6311
- asl-tek,cs5263
- chrontel,ch7218a
- dumb-vga-dac
- mstar,tsumu88adt3-lf-1
- parade,ps185hdm
@@ -81,7 +81,7 @@ examples:
port@1 {
reg = <1>;
lvds_ep: endpoint {
remote-endpoint = <&panel_ep>;
remote-endpoint = <&panel_ep>;
};
};
};
@@ -161,9 +161,9 @@ examples:
dsi0_out: endpoint {
remote-endpoint = <&d2l_in_test>;
};
};
};
};
};
};
};
- |
/* For dual-link LVDS display panel */
@@ -227,7 +227,7 @@ examples:
dsi0_out_dual: endpoint {
remote-endpoint = <&d2l_in_dual>;
};
};
};
};
};
};
};
...
@@ -54,7 +54,7 @@ examples:
display-controller@6a000000 {
compatible = "faraday,tve200";
reg = <0x6a000000 0x1000>;
interrupts = <13 IRQ_TYPE_EDGE_RISING>;
interrupts = <13 IRQ_TYPE_LEVEL_HIGH>;
resets = <&syscon GEMINI_RESET_TVC>;
clocks = <&syscon GEMINI_CLK_GATE_TVC>,
<&syscon GEMINI_CLK_TVC>;
@@ -68,9 +68,19 @@ properties:
maxItems: 1
port:
$ref: /schemas/graph.yaml#/properties/port
$ref: /schemas/graph.yaml#/$defs/port-base
unevaluatedProperties: false
description: The LCDIF output port
properties:
endpoint:
$ref: /schemas/media/video-interfaces.yaml
unevaluatedProperties: false
properties:
bus-width:
enum: [ 16, 18, 24 ]
display:
$ref: /schemas/types.yaml#/definitions/phandle
description: phandle to display panel
@@ -74,5 +74,5 @@ examples:
dc-gpios = <&gpio0 25 GPIO_ACTIVE_HIGH>;
rotation = <90>;
backlight = <&backlight>;
};
};
};
@@ -110,7 +110,7 @@ examples:
port {
parallel_out: endpoint {
remote-endpoint = <&panel_in>;
remote-endpoint = <&panel_in>;
};
};
};
@@ -150,44 +150,44 @@ examples:
/* Using an of-graph endpoint link to connect the panel */
lvds-channel@0 {
reg = <0>;
#address-cells = <1>;
#size-cells = <0>;
port@0 {
reg = <0>;
#address-cells = <1>;
#size-cells = <0>;
port@0 {
reg = <0>;
endpoint {
remote-endpoint = <&ipu_di0_lvds0>;
};
endpoint {
remote-endpoint = <&ipu_di0_lvds0>;
};
};
port@2 {
reg = <2>;
port@2 {
reg = <2>;
endpoint {
remote-endpoint = <&panel_in>;
};
};
endpoint {
remote-endpoint = <&panel_in>;
};
};
};
/* Using display-timings and fsl,data-mapping/width instead */
lvds-channel@1 {
reg = <1>;
#address-cells = <1>;
#size-cells = <0>;
fsl,data-mapping = "spwg";
fsl,data-width = <24>;
display-timings {/* ... */
};
port@1 {
reg = <1>;
#address-cells = <1>;
#size-cells = <0>;
fsl,data-mapping = "spwg";
fsl,data-width = <24>;
display-timings {/* ... */
};
port@1 {
reg = <1>;
endpoint {
remote-endpoint = <&ipu_di1_lvds1>;
};
endpoint {
remote-endpoint = <&ipu_di1_lvds1>;
};
};
};
};
@@ -158,51 +158,51 @@ examples:
#size-cells = <2>;
hdr-engine@1c114000 {
compatible = "mediatek,mt8195-disp-ethdr";
reg = <0 0x1c114000 0 0x1000>,
<0 0x1c115000 0 0x1000>,
<0 0x1c117000 0 0x1000>,
<0 0x1c119000 0 0x1000>,
<0 0x1c11a000 0 0x1000>,
<0 0x1c11b000 0 0x1000>,
<0 0x1c11c000 0 0x1000>;
reg-names = "mixer", "vdo_fe0", "vdo_fe1", "gfx_fe0", "gfx_fe1",
"vdo_be", "adl_ds";
mediatek,gce-client-reg = <&gce0 SUBSYS_1c11XXXX 0x4000 0x1000>,
<&gce0 SUBSYS_1c11XXXX 0x5000 0x1000>,
<&gce0 SUBSYS_1c11XXXX 0x7000 0x1000>,
<&gce0 SUBSYS_1c11XXXX 0x9000 0x1000>,
<&gce0 SUBSYS_1c11XXXX 0xa000 0x1000>,
<&gce0 SUBSYS_1c11XXXX 0xb000 0x1000>,
<&gce0 SUBSYS_1c11XXXX 0xc000 0x1000>;
clocks = <&vdosys1 CLK_VDO1_DISP_MIXER>,
<&vdosys1 CLK_VDO1_HDR_VDO_FE0>,
<&vdosys1 CLK_VDO1_HDR_VDO_FE1>,
<&vdosys1 CLK_VDO1_HDR_GFX_FE0>,
<&vdosys1 CLK_VDO1_HDR_GFX_FE1>,
<&vdosys1 CLK_VDO1_HDR_VDO_BE>,
<&vdosys1 CLK_VDO1_26M_SLOW>,
<&vdosys1 CLK_VDO1_HDR_VDO_FE0_DL_ASYNC>,
<&vdosys1 CLK_VDO1_HDR_VDO_FE1_DL_ASYNC>,
<&vdosys1 CLK_VDO1_HDR_GFX_FE0_DL_ASYNC>,
<&vdosys1 CLK_VDO1_HDR_GFX_FE1_DL_ASYNC>,
<&vdosys1 CLK_VDO1_HDR_VDO_BE_DL_ASYNC>,
<&topckgen CLK_TOP_ETHDR>;
clock-names = "mixer", "vdo_fe0", "vdo_fe1", "gfx_fe0", "gfx_fe1",
"vdo_be", "adl_ds", "vdo_fe0_async", "vdo_fe1_async",
"gfx_fe0_async", "gfx_fe1_async","vdo_be_async",
"ethdr_top";
power-domains = <&spm MT8195_POWER_DOMAIN_VDOSYS1>;
iommus = <&iommu_vpp M4U_PORT_L3_HDR_DS>,
<&iommu_vpp M4U_PORT_L3_HDR_ADL>;
interrupts = <GIC_SPI 517 IRQ_TYPE_LEVEL_HIGH 0>; /* disp mixer */
resets = <&vdosys1 MT8195_VDOSYS1_SW1_RST_B_HDR_VDO_FE0_DL_ASYNC>,
<&vdosys1 MT8195_VDOSYS1_SW1_RST_B_HDR_VDO_FE1_DL_ASYNC>,
<&vdosys1 MT8195_VDOSYS1_SW1_RST_B_HDR_GFX_FE0_DL_ASYNC>,
<&vdosys1 MT8195_VDOSYS1_SW1_RST_B_HDR_GFX_FE1_DL_ASYNC>,
<&vdosys1 MT8195_VDOSYS1_SW1_RST_B_HDR_VDO_BE_DL_ASYNC>;
reset-names = "vdo_fe0_async", "vdo_fe1_async", "gfx_fe0_async",
"gfx_fe1_async", "vdo_be_async";
compatible = "mediatek,mt8195-disp-ethdr";
reg = <0 0x1c114000 0 0x1000>,
<0 0x1c115000 0 0x1000>,
<0 0x1c117000 0 0x1000>,
<0 0x1c119000 0 0x1000>,
<0 0x1c11a000 0 0x1000>,
<0 0x1c11b000 0 0x1000>,
<0 0x1c11c000 0 0x1000>;
reg-names = "mixer", "vdo_fe0", "vdo_fe1", "gfx_fe0", "gfx_fe1",
"vdo_be", "adl_ds";
mediatek,gce-client-reg = <&gce0 SUBSYS_1c11XXXX 0x4000 0x1000>,
<&gce0 SUBSYS_1c11XXXX 0x5000 0x1000>,
<&gce0 SUBSYS_1c11XXXX 0x7000 0x1000>,
<&gce0 SUBSYS_1c11XXXX 0x9000 0x1000>,
<&gce0 SUBSYS_1c11XXXX 0xa000 0x1000>,
<&gce0 SUBSYS_1c11XXXX 0xb000 0x1000>,
<&gce0 SUBSYS_1c11XXXX 0xc000 0x1000>;
clocks = <&vdosys1 CLK_VDO1_DISP_MIXER>,
<&vdosys1 CLK_VDO1_HDR_VDO_FE0>,
<&vdosys1 CLK_VDO1_HDR_VDO_FE1>,
<&vdosys1 CLK_VDO1_HDR_GFX_FE0>,
<&vdosys1 CLK_VDO1_HDR_GFX_FE1>,
<&vdosys1 CLK_VDO1_HDR_VDO_BE>,
<&vdosys1 CLK_VDO1_26M_SLOW>,
<&vdosys1 CLK_VDO1_HDR_VDO_FE0_DL_ASYNC>,
<&vdosys1 CLK_VDO1_HDR_VDO_FE1_DL_ASYNC>,
<&vdosys1 CLK_VDO1_HDR_GFX_FE0_DL_ASYNC>,
<&vdosys1 CLK_VDO1_HDR_GFX_FE1_DL_ASYNC>,
<&vdosys1 CLK_VDO1_HDR_VDO_BE_DL_ASYNC>,
<&topckgen CLK_TOP_ETHDR>;
clock-names = "mixer", "vdo_fe0", "vdo_fe1", "gfx_fe0", "gfx_fe1",
"vdo_be", "adl_ds", "vdo_fe0_async", "vdo_fe1_async",
"gfx_fe0_async", "gfx_fe1_async","vdo_be_async",
"ethdr_top";
power-domains = <&spm MT8195_POWER_DOMAIN_VDOSYS1>;
iommus = <&iommu_vpp M4U_PORT_L3_HDR_DS>,
<&iommu_vpp M4U_PORT_L3_HDR_ADL>;
interrupts = <GIC_SPI 517 IRQ_TYPE_LEVEL_HIGH 0>; /* disp mixer */
resets = <&vdosys1 MT8195_VDOSYS1_SW1_RST_B_HDR_VDO_FE0_DL_ASYNC>,
<&vdosys1 MT8195_VDOSYS1_SW1_RST_B_HDR_VDO_FE1_DL_ASYNC>,
<&vdosys1 MT8195_VDOSYS1_SW1_RST_B_HDR_GFX_FE0_DL_ASYNC>,
<&vdosys1 MT8195_VDOSYS1_SW1_RST_B_HDR_GFX_FE1_DL_ASYNC>,
<&vdosys1 MT8195_VDOSYS1_SW1_RST_B_HDR_VDO_BE_DL_ASYNC>;
reset-names = "vdo_fe0_async", "vdo_fe1_async", "gfx_fe0_async",
"gfx_fe1_async", "vdo_be_async";
};
};
...
@@ -112,24 +112,24 @@ examples:
mediatek,syscon-hdmi = <&mmsys 0x900>;
ports {
#address-cells = <1>;
#size-cells = <0>;
#address-cells = <1>;
#size-cells = <0>;
port@0 {
reg = <0>;
port@0 {
reg = <0>;
hdmi0_in: endpoint {
remote-endpoint = <&dpi0_out>;
hdmi0_in: endpoint {
remote-endpoint = <&dpi0_out>;
};
};
};
port@1 {
reg = <1>;
port@1 {
reg = <1>;
hdmi0_out: endpoint {
remote-endpoint = <&hdmi_con_in>;
hdmi0_out: endpoint {
remote-endpoint = <&hdmi_con_in>;
};
};
};
};
};
@@ -305,7 +305,9 @@ allOf:
properties:
compatible:
contains:
const: qcom,adreno-gmu-810.0
enum:
- qcom,adreno-gmu-810.0
- qcom,adreno-gmu-830.1
then:
properties:
reg:
@@ -363,7 +365,9 @@ allOf:
properties:
compatible:
contains:
const: qcom,adreno-gmu-x285.1
enum:
- qcom,adreno-gmu-x285.0
- qcom,adreno-gmu-x290.1
then:
properties:
reg:
@@ -414,6 +414,7 @@ allOf:
contains:
enum:
- qcom,adreno-44010000
- qcom,adreno-44050001
- qcom,adreno-44050a01
- qcom,adreno-44070001
then:
@@ -451,7 +452,9 @@ allOf:
- qcom,adreno-43050c01
- qcom,adreno-43051401
- qcom,adreno-44010000
- qcom,adreno-44050001
- qcom,adreno-44050a01
- qcom,adreno-44060000
- qcom,adreno-44070001
then: # Starting with A6xx, the clocks are usually defined in the GMU node
@@ -113,7 +113,7 @@ allOf:
- const: core
- const: master_iface
- const: slave_iface
core-vcc-supplies: false
core-vcc-supply: false
- if:
properties:
@@ -136,7 +136,7 @@ allOf:
- const: core
- const: alt_iface
- const: extp
hdmi-mux-supplies: false
hdmi-mux-supply: false
- if:
properties:
@@ -26,6 +26,7 @@ properties:
- qcom,msm8916-mdp5
- qcom,msm8917-mdp5
- qcom,msm8937-mdp5
- qcom,msm8952-mdp5
- qcom,msm8953-mdp5
- qcom,msm8974-mdp5
- qcom,msm8976-mdp5
@@ -14,10 +14,15 @@ allOf:
properties:
compatible:
items:
- enum:
- tianma,tl121bvms07-00
- const: ilitek,il79900a
oneOf:
- items:
- enum:
- raspberrypi,dsi-10inch
- const: ilitek,il79600a
- items:
- enum:
- tianma,tl121bvms07-00
- const: ilitek,il79900a
reg:
maxItems: 1
@@ -0,0 +1,58 @@
# SPDX-License-Identifier: GPL-2.0-only OR BSD-2-Clause
%YAML 1.2
---
$id: http://devicetree.org/schemas/display/panel/ilitek,ili7836a.yaml#
$schema: http://devicetree.org/meta-schemas/core.yaml#
title: Ilitek ILI7836A-based OLED panels
maintainers:
- Neil Armstrong <[email protected]>
allOf:
- $ref: panel-common.yaml#
properties:
compatible:
items:
- const: retroidpocket,nova-panel
- const: ilitek,ili7836a
reg:
maxItems: 1
description: DSI virtual channel
vddio-supply: true
avdd-supply: true
backlight: true
port: true
reset-gpios: true
rotation: true
required:
- compatible
- reg
- vddio-supply
- avdd-supply
- reset-gpios
additionalProperties: false
examples:
- |
#include <dt-bindings/gpio/gpio.h>
dsi {
#address-cells = <1>;
#size-cells = <0>;
panel@0 {
compatible = "retroidpocket,nova-panel", "ilitek,ili7836a";
reg = <0>;
vddio-supply = <&vdd_disp_1v8>;
avdd-supply = <&vdd_disp2_2v8>;
reset-gpios = <&tlmm 133 GPIO_ACTIVE_HIGH>;
};
};
...
@@ -0,0 +1,83 @@
# SPDX-License-Identifier: GPL-2.0-only OR BSD-2-Clause
%YAML 1.2
---
$id: http://devicetree.org/schemas/display/panel/novatek,nt36532.yaml#
$schema: http://devicetree.org/meta-schemas/core.yaml#
title: Novatek NT36532 based DSI display Panels
maintainers:
- Junjie Cao <[email protected]>
description: |
The NT36532 IC from Novatek is a Driver IC used to drive MIPI-DSI panels.
allOf:
- $ref: panel-common-dual.yaml#
properties:
compatible:
items:
- enum:
- csot,ppc100hb1-1
- const: novatek,nt36532
reg:
maxItems: 1
reset-gpios:
maxItems: 1
description: phandle of gpio for reset line - This should be 8mA
vddio-supply:
description: regulator that supplies the I/O voltage
ports: true
required:
- compatible
- reg
- vddio-supply
- reset-gpios
- ports
unevaluatedProperties: false
examples:
- |
#include <dt-bindings/gpio/gpio.h>
dsi {
#address-cells = <1>;
#size-cells = <0>;
panel@0 {
compatible = "csot,ppc100hb1-1", "novatek,nt36532";
reg = <0>;
reset-gpios = <&tlmm 133 GPIO_ACTIVE_LOW>;
vddio-supply = <&vreg_l12b_1p8>;
ports {
#address-cells = <1>;
#size-cells = <0>;
port@0 {
reg = <0>;
panel_in_0: endpoint {
remote-endpoint = <&dsi0_out>;
};
};
port@1 {
reg = <1>;
panel_in_1: endpoint {
remote-endpoint = <&dsi1_out>;
};
};
};
};
};
...
@@ -0,0 +1,87 @@
# SPDX-License-Identifier: GPL-2.0-only OR BSD-2-Clause
%YAML 1.2
---
$id: http://devicetree.org/schemas/display/panel/novatek,nt51021.yaml#
$schema: http://devicetree.org/meta-schemas/core.yaml#
title: Novatek NT51021 based DSI display Panels
maintainers:
- Nickolay Goppen <[email protected]>
description:
The NT51021 IC from Novatek is a generic DSI Panel IC used to drive dsi
panels.
Right now, support is added only for a BOE WUXGA LCD display panels with a
resolution of 1200x1920. They are video mode DSI panels.
allOf:
- $ref: panel-common.yaml#
properties:
compatible:
items:
- enum:
- boe,tv101wum-nm0
- boe,tv101wum-nx0
- boe,tv080wum-nx2
- const: novatek,nt51021
reg:
maxItems: 1
reset-gpios:
maxItems: 1
description: phandle of gpio for reset line - This should be 8mA, gpio
can be configured using mux, pinctrl, pinctrl-names (active low)
vdd-supply:
description: phandle of the regulator that provides the supply voltage
for digital circuits
avdd-supply:
description: phandle of the regulator that provides the supply voltage
for analog circuits
havdd-supply:
description: phandle of the regulator that provides the supply voltage
for analog circuits
required:
- compatible
- reg
- vdd-supply
- avdd-supply
- havdd-supply
- reset-gpios
- port
unevaluatedProperties: false
examples:
- |
#include <dt-bindings/gpio/gpio.h>
dsi {
#address-cells = <1>;
#size-cells = <0>;
panel@0 {
compatible = "boe,tv101wum-nx0", "novatek,nt51021";
reg = <0>;
vdd-supply = <&lcd_pwr>;
avdd-supply = <&lcd_pwr>;
havdd-supply = <&lcd_pwr>;
backlight = <&i2c_backlight>;
reset-gpios = <&tlmm 62 GPIO_ACTIVE_LOW>;
port {
panel_in: endpoint {
remote-endpoint = <&dsi0_out>;
};
};
};
};
...
@@ -44,6 +44,8 @@ properties:
- auo,t215hvn01
# BOE AV123Z7M-N17 12.3" (1920x720) LVDS TFT LCD panel
- boe,av123z7m-n17
# BOE DV215FHM-R01 21.5" FHD (1920x1080) dual-channel LVDS panel
- boe,dv215fhm-r01
# InnoLux 15.6" FHD (1920x1080) TFT LCD panel
- innolux,g156hce-l01
# Kaohsiung Opto-Electronics Inc. 10.1" WUXGA (1920 x 1200) LVDS TFT LCD panel
@@ -338,6 +338,8 @@ properties:
- tpk,f07a-0102
# TPK U.S.A. LLC Fusion 10.1" 1024 x 600 (WSVGA) LCD panel with capacitive touch
- tpk,f10a-0102
# Truly Semiconductors Limited 3.5" QVGA (320 x 240) TFT LCD panel
- truly,tft320240-229-e
# United Radiant Technology UMSH-8596MD-xT 7.0" WVGA TFT LCD panel
# Supported are LVDS versions (-11T, -19T) and parallel ones
# (-T, -1T, -7T, -20T).
@@ -0,0 +1,74 @@
# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
%YAML 1.2
---
$id: http://devicetree.org/schemas/display/panel/raydium,rm69220.yaml#
$schema: http://devicetree.org/meta-schemas/core.yaml#
title: Raydium RM69220 based MIPI-DSI panels
maintainers:
- Alexander Koskovich <[email protected]>
description:
The Raydium RM69220 is a generic DSI Panel IC used to control
AMOLED panels.
allOf:
- $ref: panel-common.yaml#
properties:
compatible:
items:
- enum:
- boe,bf068mwm-td0
- const: raydium,rm69220
reg:
maxItems: 1
dvdd-supply:
description: Digital voltage rail
vci-supply:
description: Analog voltage rail
vddio-supply:
description: I/O voltage rail
required:
- compatible
- reg
- dvdd-supply
- vci-supply
- vddio-supply
- reset-gpios
- port
unevaluatedProperties: false
examples:
- |
#include <dt-bindings/gpio/gpio.h>
dsi {
#address-cells = <1>;
#size-cells = <0>;
panel@0 {
compatible = "boe,bf068mwm-td0", "raydium,rm69220";
reg = <0>;
reset-gpios = <&tlmm 12 GPIO_ACTIVE_LOW>;
dvdd-supply = <&vreg_oled_dvdd_1p8>;
vci-supply = <&vreg_l19b>;
vddio-supply = <&vreg_l9b>;
port {
panel_in_0: endpoint {
remote-endpoint = <&dsi0_out>;
};
};
};
};
...
@@ -51,7 +51,7 @@ examples:
port {
mipi_in_panel: endpoint {
remote-endpoint = <&mipi_out_panel>;
remote-endpoint = <&mipi_out_panel>;
};
};
};
@@ -34,7 +34,7 @@ required:
- compatible
- reg
additionalProperties: false
unevaluatedProperties: false
examples:
- |
@@ -14,7 +14,11 @@ allOf:
properties:
compatible:
const: visionox,vtdr6130
oneOf:
- const: visionox,vtdr6130
- items:
- const: retroidpocket,rp6-panel
- const: visionox,vtdr6130
reg:
maxItems: 1
@@ -25,13 +29,13 @@ properties:
vdd-supply: true
port: true
reset-gpios: true
rotation: true
required:
- compatible
- reg
- vddio-supply
- vci-supply
- vdd-supply
- reset-gpios
- port
@@ -160,6 +160,7 @@ examples:
reg = <1>;
edp_out_panel: endpoint {
data-lanes = <0 1>;
remote-endpoint = <&panel_in_edp>;
};
};
@@ -64,5 +64,5 @@ examples:
vfront-porch = <0>;
vsync-len = <0>;
};
};
};
};
};
@@ -0,0 +1,42 @@
# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
%YAML 1.2
---
$id: http://devicetree.org/schemas/display/solomon,ssd1351.yaml#
$schema: http://devicetree.org/meta-schemas/core.yaml#
title: Solomon SSD1351 OLED Display Controller
maintainers:
- Amit Barzilai <[email protected]>
- Javier Martinez Canillas <[email protected]>
allOf:
- $ref: solomon,ssd-common.yaml#
properties:
compatible:
enum:
- solomon,ssd1351
required:
- compatible
- reg
unevaluatedProperties: false
examples:
- |
#include <dt-bindings/gpio/gpio.h>
spi {
#address-cells = <1>;
#size-cells = <0>;
oled@0 {
compatible = "solomon,ssd1351";
reg = <0x0>;
reset-gpios = <&gpio2 7 GPIO_ACTIVE_LOW>;
dc-gpios = <&gpio2 8 GPIO_ACTIVE_HIGH>;
spi-max-frequency = <10000000>;
};
};
@@ -101,35 +101,35 @@ examples:
#size-cells = <0>;
ports {
#address-cells = <1>;
#size-cells = <0>;
#address-cells = <1>;
#size-cells = <0>;
port@0 {
reg = <0>;
dsi_in: endpoint {
remote-endpoint = <&ltdc_ep1_out>;
};
};
port@0 {
reg = <0>;
dsi_in: endpoint {
remote-endpoint = <&ltdc_ep1_out>;
};
};
port@1 {
reg = <1>;
dsi_out: endpoint {
remote-endpoint = <&panel_in>;
};
};
port@1 {
reg = <1>;
dsi_out: endpoint {
remote-endpoint = <&panel_in>;
};
};
};
panel@0 {
compatible = "orisetech,otm8009a";
reg = <0>;
reset-gpios = <&gpioe 4 GPIO_ACTIVE_LOW>;
power-supply = <&v3v3>;
compatible = "orisetech,otm8009a";
reg = <0>;
reset-gpios = <&gpioe 4 GPIO_ACTIVE_LOW>;
power-supply = <&v3v3>;
port {
panel_in: endpoint {
remote-endpoint = <&dsi_out>;
};
};
port {
panel_in: endpoint {
remote-endpoint = <&dsi_out>;
};
};
};
};
@@ -119,9 +119,9 @@ examples:
resets = <&rcc LTDC_R>;
port {
ltdc_out_dsi: endpoint {
remote-endpoint = <&dsi_in>;
};
ltdc_out_dsi: endpoint {
remote-endpoint = <&dsi_in>;
};
};
};
@@ -114,14 +114,14 @@ examples:
port@0 {
reg = <0>;
lvds_in: endpoint {
remote-endpoint = <&ltdc_ep1_out>;
remote-endpoint = <&ltdc_ep1_out>;
};
};
port@1 {
reg = <1>;
lvds_out0: endpoint {
remote-endpoint = <&lvds_panel_in>;
remote-endpoint = <&lvds_panel_in>;
};
};
};
@@ -35,6 +35,9 @@ properties:
resets:
maxItems: 1
power-domains:
maxItems: 1
epod-supply:
description: a phandle to the EPOD regulator
@@ -109,6 +112,7 @@ examples:
- |
#include <dt-bindings/interrupt-controller/irq.h>
#include <dt-bindings/interrupt-controller/arm-gic.h>
#include <dt-bindings/arm/ux500_pm_domains.h>
#include <dt-bindings/mfd/dbx500-prcmu.h>
#include <dt-bindings/gpio/gpio.h>
@@ -116,6 +120,7 @@ examples:
compatible = "ste,mcde";
reg = <0xa0350000 0x1000>;
interrupts = <GIC_SPI 48 IRQ_TYPE_LEVEL_HIGH>;
power-domains = <&pm_domains DOMAIN_B2R2_MCDE>;
epod-supply = <&db8500_b2r2_mcde_reg>;
vana-supply = <&ab8500_ldo_ana_reg>;
clocks = <&prcmu_clk PRCMU_MCDECLK>,
@@ -98,6 +98,18 @@ properties:
items:
- description: phandle to the HEG or core power domain
nvidia,channels:
$ref: /schemas/types.yaml#/definitions/uint32-array
minItems: 2
maxItems: 2
description: Describes range of usable Host1x channels as a <base count> tuple.
nvidia,syncpoints:
$ref: /schemas/types.yaml#/definitions/uint32-array
minItems: 2
maxItems: 2
description: Describes range of writable Host1x syncpoints as a <base count> tuple.
required:
- compatible
- interrupts
@@ -147,14 +159,20 @@ allOf:
then:
properties:
reg-names:
items:
- const: hypervisor
- const: vm
oneOf:
- items:
- const: hypervisor
- const: vm
- items:
- const: vm
reg:
items:
- description: region used by the hypervisor
- description: region assigned to the virtual machine
oneOf:
- items:
- description: region used by the hypervisor
- description: region assigned to the virtual machine
- items:
- description: region assigned to the virtual machine
resets:
maxItems: 1
@@ -198,16 +216,22 @@ allOf:
then:
properties:
reg-names:
items:
- const: common
- const: hypervisor
- const: vm
oneOf:
- items:
- const: common
- const: hypervisor
- const: vm
- items:
- const: vm
reg:
items:
- description: region used by host1x server
- description: region used by the hypervisor
- description: region assigned to the virtual machine
oneOf:
- items:
- description: region used by host1x server
- description: region used by the hypervisor
- description: region assigned to the virtual machine
- items:
- description: region assigned to the virtual machine
interrupts:
items:
+1
View File
@@ -23,4 +23,5 @@ Next (GCN), Radeon DNA (RDNA), and Compute DNA (CDNA) architectures.
debugfs
process-isolation
amdgpu-glossary
ualink
ptl
+75
View File
@@ -0,0 +1,75 @@
==============
UALink Support
==============
Overview
========
Connected GPUs in a pod can directly access the remove memory on another GPU
over UALink. Unlike RMDA, there is no copy involved; it is direct loads/stores
over the fabric. Shared memory can only be accessed by a remote GPU if the
memory was exported and the importer has been authorized. For the memory to be
shared, it must be part of a unified physical address space shared between
nodes. This address space is called NPA (Nework Physical Address) space. This
address space is partitioned between the GPUs so that each GPU has its own
segment of the address space in which to export its memory. Each GPU maintains
a dedicated set of page tables for their NPA space similar to GPUVM. Note that
this mechanism only allows for GPU access to remote memory. The remote memory
is not CPU accessible.
Exported memory is pinned. This is similar to how dma-bufs in VRAM are pinned
for P2P access. Remote TLB shootdowns from the exporter are used when the
exported memory is freed in order to remove access by remote GPUs.
To access remote memory, the driver can map NPA addresses into its per process
GPUVM page tables just like local memory. Applications use opaque handles to
represent remote memory. GPUs in a pod communicate with eachother directly to
exchange NPA addresses between importers and exporters. If a node goes offline
or is reset, their peers will clean up any remaining refrences that are lost
when that happens. NPA addresses are exchanged directly between the kernel
drivers using the scale up fabric. NPA addresses are never exposed to user
space.
On the importer, the NPA space is like another physical address space. NPA
addresses can be used as physical addresses for GPUVM to provide GPU virtual
addresses to the memory for processes using the GPU.
On the exporter, the NPA space provides a way to expose discontiguous local
memory as a contiguous address range for remote GPUs. This allows the exporter
to locally manage the pages mapped into the NPA space.
Remote NPAs are managed like another device specific TTM pool similar to
doorbells or VRAM, however they cannot be CPU mapped.
User Interface
==============
Two IOCTLs are provided to export and import remote memory.
Export Memory
-------------
To export memory, a UALINK handle must be created for an allocation that can be
shared with another node in the pod. To do this the exporter calls the GEM
UALink IOCTL with the GEM handle to the buffer it wants to export. The IOCTL
returns a unique 128 bit handle which can be shared with the remote host.
Calling export on the same GEM handle always returns the same UALink handle.
The UALink handle is destroyed when the GEM object reference count reaches 0.
Import Memory
-------------
To import remote memory, the UALink handle from the remote node must be
converted from a UALink handle to a local GEM object which represents the local
reference to the NPA space on the importer. If the memory has already been
imported, it just returns a new reference to the existing GEM object. If not,
the importer queries the exporter to get the NPA address. Once it has that, the
importer can create the GEM to represent the NPA space used by the allocation.
The GEM object is then exported to the caller as a dma-buf. The dma-buf is
leveraged for dynamic attachment which provides the ability to revoke access
when necessary.
Device to Device Communications
===============================
Devices communicate via a protocol implemented in firmware. Mesages sent to a
remote node generate an interrupt on that node for servicing.
+14 -5
View File
@@ -109,21 +109,30 @@ VBLANK Helper Reference
fbdev Helper Functions Reference
================================
.. kernel-doc:: drivers/gpu/drm/drm_fb_helper.c
.. kernel-doc:: drivers/gpu/drm/clients/drm_fbdev_helper.c
:doc: fbdev helpers
.. kernel-doc:: include/drm/drm_fb_helper.h
.. kernel-doc:: include/drm/clients/drm_fbdev_helper.h
:internal:
.. kernel-doc:: drivers/gpu/drm/drm_fb_helper.c
.. kernel-doc:: drivers/gpu/drm/clients/drm_fbdev_helper.c
:export:
format Helper Functions Reference
Draw Helper Functions Reference
=================================
Format-Conversion Helper Reference
----------------------------------
.. kernel-doc:: drivers/gpu/drm/drm_format_helper.c
:export:
Panic Helper Reference
----------------------
.. kernel-doc:: drivers/gpu/drm/drm_panic_helper.c
:export:
Framebuffer DMA Helper Functions Reference
==========================================
@@ -209,7 +218,7 @@ Bridge Connector Helper Reference
Panel-Bridge Helper Reference
-----------------------------
.. kernel-doc:: drivers/gpu/drm/bridge/panel.c
.. kernel-doc:: drivers/gpu/drm/drm_panel.c
:export:
.. _drm_panel_helper:
+3
View File
@@ -421,6 +421,9 @@ Plane Panic Functions Reference
.. kernel-doc:: include/drm/drm_panic.h
:internal:
.. kernel-doc:: drivers/gpu/drm/drm_panic_internal.h
:internal:
.. kernel-doc:: drivers/gpu/drm/drm_panic.c
:export:
+39
View File
@@ -56,6 +56,11 @@ User space tools can:
``node-id`` and ``error-id`` as parameters.
* Clear specific error counters with the ``clear-error-counter`` command, using both
``node-id`` and ``error-id`` as parameters.
* Subscribe to the ``error-report`` multicast group to receive ``error-event``.
* Query specific error counter threshold with the ``get-error-threshold`` command, using both
``node-id`` and ``error-id`` as parameters.
* Set specific error counter threshold with the ``set-error-threshold`` command, using
``node-id``, ``error-id`` and ``error-threshold`` as parameters.
YAML-based Interface
--------------------
@@ -111,3 +116,37 @@ Example: Clear an error counter for a given node
sudo ynl --family drm_ras --do clear-error-counter --json '{"node-id":0, "error-id":1}'
None
Example: Subscribe to ``error-report`` multicast group
.. code-block:: bash
sudo ynl --family drm_ras --output-json --subscribe error-report
.. code-block:: json
{
"name": "error-event",
"msg": {
"device-name": "0000:03:00.0",
"node-id": 1,
"node-name": "uncorrectable-errors",
"error-id": 1,
"error-name": "error_name1",
"error-value": 1
}
}
Example: Query error threshold of a given counter
.. code-block:: bash
sudo ynl --family drm_ras --do get-error-threshold --json '{"node-id":0, "error-id":1}'
{'error-id': 1, 'error-name': 'error_name1', 'error-threshold': 16}
Example: Set error threshold of a given counter
.. code-block:: bash
sudo ynl --family drm_ras --do set-error-threshold --json '{"node-id":0, "error-id":1, "error-threshold":8}'
None
+88 -5
View File
@@ -424,7 +424,7 @@ needed.
Recovery
--------
Current implementation defines four recovery methods, out of which, drivers
Current implementation defines several recovery methods, out of which, drivers
can use any one, multiple or none. Method(s) of choice will be sent in the
uevent environment as ``WEDGED=<method1>[,..,<methodN>]`` in order of less to
more side-effects. See the section `Vendor Specific Recovery`_
@@ -434,15 +434,16 @@ method is unknown, ``WEDGED=unknown`` will be sent instead.
Userspace consumers can parse this event and attempt recovery as per the
following expectations.
=============== ========================================
=============== =========================================
Recovery method Consumer expectations
=============== ========================================
=============== =========================================
none optional telemetry collection
rebind unbind + bind driver
bus-reset unbind + bus reset/re-enumeration + bind
vendor-specific vendor specific recovery method
cold-reset remove device + slot power cycle + rescan
unknown consumer policy
=============== ========================================
=============== =========================================
No Recovery
-----------
@@ -453,6 +454,17 @@ debug purpose in order to root cause the hang. This is useful because the first
hang is usually the most critical one which can result in consequential hangs
or complete wedging.
Cold Reset Recovery
-------------------
When ``WEDGED=cold-reset`` is sent, it indicates that the device has
encountered an error condition that cannot be resolved through other
recovery methods such as driver rebind or bus reset, and requires a complete
device power cycle to restore functionality.
This method is used by devices that are plugged directly into the PCIe slot
which supports removing the power.
Vendor Specific Recovery
------------------------
@@ -516,7 +528,7 @@ Example - rebind
Udev rule::
SUBSYSTEM=="drm", ENV{WEDGED}=="rebind", DEVPATH=="*/drm/card[0-9]",
SUBSYSTEM=="drm", ENV{WEDGED}=="rebind", DEVPATH=="*/drm/card[0-9]", \
RUN+="/path/to/rebind.sh $env{DEVPATH}"
Recovery script::
@@ -530,6 +542,77 @@ Recovery script::
echo -n $DEVICE > $DRIVER/unbind
echo -n $DEVICE > $DRIVER/bind
Example - cold-reset
--------------------
Udev rule::
SUBSYSTEM=="drm", ENV{WEDGED}=="cold-reset", DEVPATH=="*/drm/card[0-9]", \
RUN+="/path/to/cold-reset.sh $env{DEVPATH}"
Recovery script::
#!/bin/sh
die() { echo "ERROR: $*" >&2; exit 1; }
[ -n "$1" ] || die "Usage: $0 <device-path>"
PCI_DEVS=/sys/bus/pci/devices
PCI_SLOTS=/sys/bus/pci/slots
syspath=$(readlink -f "/sys/$1/device" 2>/dev/null || readlink -f "/sys/$1" 2>/dev/null)
[ -n "$syspath" ] || die "cannot resolve sysfs path for: $1"
dev=$(basename "$syspath")
[ -e "$PCI_DEVS/$dev" ] || die "not a PCI device: $dev"
echo "device : $dev"
slot=""
walk=$(dirname "$(readlink -f "$PCI_DEVS/$dev")")
while true; do
ancestor=$(basename "$walk")
case "$ancestor" in pci*) break ;; esac # reached the virtual bus root
ancestor_nofn=${ancestor%.*} # strip function: 0000:03:01.0 -> 0000:03:01
for f in "$PCI_SLOTS"/*/address; do
[ -f "$f" ] || continue
addr=$(cat "$f")
case "$ancestor_nofn" in
*"$addr") slot=$(basename "$(dirname "$f")"); break ;;
esac
done
if [ -n "$slot" ] && [ -e "$PCI_SLOTS/$slot/power" ]; then
echo "slot : $slot (port $ancestor)"
break
fi
slot=""
walk=$(dirname "$walk")
done
[ -n "$slot" ] || die "no hotplug slot with power control found in PCIe topology"
# Cold reset: remove the device, cut slot power, restore power, rescan.
echo "Removing $dev..."
[ -e "$PCI_DEVS/$dev" ] && echo 1 > "$PCI_DEVS/$dev/remove"
echo "Powering off slot $slot..."
echo 0 > "$PCI_SLOTS/$slot/power"
echo "Powering on slot $slot..."
echo 1 > "$PCI_SLOTS/$slot/power"
echo "Rescanning PCI bus..."
echo 1 > /sys/bus/pci/rescan
if [ -e "$PCI_DEVS/$dev" ]; then
echo "Done: $dev is back online."
else
echo "WARNING: $dev did not re-appear after rescan."
fi
Customization
-------------
+2
View File
@@ -106,6 +106,8 @@ All FSP messages share a common header format consisting of two 32-bit words:
- Bit 30: EOM (End of Message)
- Bits 29:28: Packet sequence number
- Bits 23:16: Source Endpoint ID
- Bits 15:8: Destination Endpoint ID
- Bits 3:0: MCTP header version (1)
**NVDM header** (NVIDIA Vendor Defined Message):
+603
View File
@@ -0,0 +1,603 @@
.. SPDX-License-Identifier: GPL-2.0
.. SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
=============================================
GPU interrupt handling: GIN and the GSP event
=============================================
This document describes how nova-core receives interrupts from the GPU on Turing
and later chipsets. It covers the GPU Interrupt and Notification unit (GIN),
which is the GPU's interrupt controller, and the GSP event, the interrupt that
nova-core services in normal operation.
Throughout, *CPU* means the CPU and the nova-core driver running on it. The GPU
also has on-chip processors that run their own firmware and receive their own
interrupts. The GSP (GPU System Processor) is one of them.
Register names are the names from the GPU hardware reference headers. The
pre-Hopper headers call the controller ``NV_CTRL`` and the Hopper-plus headers
call it ``NV_GIN``. This document calls it GIN throughout, because the tree that
nova-core services is the same on every supported chipset. OpenRM, NVIDIA's
open-source GPU kernel driver, is cited wherever nova-core follows it.
Terminology
===========
The three levels of the controller, innermost first:
leaf
One ``LEAF`` register. Each of its 32 bits is the pending bit of one
interrupt source. A Turing, Ampere, or Ada tree has 8 leaves. A Hopper or
Blackwell tree has 16.
subtree
Two consecutive leaves, summarized by one bit of ``TOP``. A subtree is the
unit of enabling at ``TOP``, and under MSI-X it is the unit of delivery:
every interrupt from one subtree arrives on one MSI-X entry.
tree
One ``TOP`` register and the leaves under it. Every PCIe function has its
own tree, and nova-core services the CPU tree of one function.
The hardware headers, OpenRM, and the Linux PCI API all use the word "vector",
each for a different number. This document gives each one its own name, and a
bare "vector" always means a GIN vector.
GIN vector
The GPU-internal interrupt source number. It addresses one bit of one leaf.
A 16-leaf tree holds vectors 0 through 511, and an 8-leaf tree holds 0
through 255. The CPU doorbell is vector 129 and the GSP event is vector
155.
MSI-X entry
An index into the device's MSI-X table. One entry serves one subtree.
PCI vector
One of the interrupts that ``pci_alloc_irq_vectors()`` allocates: an MSI-X
entry, or the single MSI message.
Linux IRQ number
What ``request_irq()`` takes, obtained from ``pci_irq_vector()`` for a PCI
vector. Linux's ``struct msix_entry`` calls this number ``.vector`` as
well.
The remaining terms, each named for the register or the specification that
defines it:
enable, disable a vector
Writes to ``LEAF_EN_SET`` and ``LEAF_EN_CLEAR``.
enable, disable a subtree
Writes to ``TOP_EN_SET`` and ``TOP_EN_CLEAR``.
serviced subtree
A subtree that nova-core enables and has a handler for.
rearm
Restoring PCI interrupt delivery after servicing an interrupt. See
"Rearming PCI interrupt delivery".
mask
Reserved for the two places where hardware and the PCI specification use
the word: the MSI-X per-entry Vector Control mask bit, which Linux
controls, and the falcon interrupt masks. It never names a GIN enable.
latched, pending
Two names for one state, a set ``LEAF`` bit. The vector's source sets the
bit whether or not the vector is enabled.
clear a vector
Write a 1 to the vector's bit in ``LEAF``. OpenRM calls the same operation
``intrClearLeafVector_HAL``.
pending bits
The plain 32-bit value read from a ``LEAF`` register.
notification
An interrupt whose only content is that something happened, such as a
posted message. Servicing a notification means reading what it announces.
The unit that raised it needs no attention. The GSP event is one.
unit
Any block that raises an interrupt. "Engine" is reserved for the blocks
that do user work: GR, CE, NVDEC, and the like.
falcon
One of the GPU's microcontrollers (see
Documentation/gpu/nova/core/falcon.rst). The GSP runs on the RISC-V core
inside its falcon. A falcon latches each of its interrupt causes and routes
it either to the host, meaning the CPU, or to its own core.
The GIN controller
==================
A GPU has many interrupt sources: the GSP, the copy engines, the graphics
engine, video decode and encode, the MMU fault path, timers, and others. GIN
records which of them are pending and raises the PCI interrupt to the CPU.
Trees
-----
GIN keeps one tree for each destination it can deliver an interrupt to. The CPU
has one tree per PCIe function, so the physical function and each virtual
function have their own. The GSP has a tree, and so do the other on-chip
processors that receive interrupts. Every tree has the same two-level layout,
and a function reaches its own tree through the per-function register aperture.
nova-core services the CPU tree of one function. A virtual function's tree
belongs to that function's driver, and a processor's tree belongs to the
firmware running on that processor.
The two-level tree
------------------
A tree is a set of ``LEAF`` registers and one ``TOP`` register.
* ``LEAF(i)`` is a 32-bit register that holds the pending bits of vectors
``32i`` through ``32i + 31``. A set bit is a pending vector.
* ``TOP`` is a 32-bit read-only register. Bit ``N`` summarizes subtree ``N``,
which is ``LEAF(2N)`` and ``LEAF(2N + 1)``. The bit is set when an enabled
vector is pending in either leaf.
A tree with L leaves has L / 2 subtrees and uses TOP bits 0 through L / 2 - 1.
The other TOP bits read 0. The leaves and subtrees that a chipset has are its
implemented leaves and subtrees, and "Per-architecture differences" gives the
counts. For an 8-leaf tree::
TOP bit 0 -> subtree 0 -> LEAF(0), LEAF(1) vectors 0..63
TOP bit 1 -> subtree 1 -> LEAF(2), LEAF(3) vectors 64..127
TOP bit 2 -> subtree 2 -> LEAF(4), LEAF(5) vectors 128..191
TOP bit 3 -> subtree 3 -> LEAF(6), LEAF(7) vectors 192..255
LEAF(4), one bit per vector, holds vectors 128..159:
bit 1 = vector 129 (CPU doorbell)
bit 27 = vector 155 (GSP event)
A vector's number fixes its place in the tree::
leaf = vector / 32
bit = vector % 32
subtree = leaf / 2
Registers
---------
nova-core defines the tree's registers in the ``irq`` module's ``regs.rs``. The
leaf registers are arrays indexed by leaf number.
* ``LEAF(i)`` reads as the pending bits of leaf ``i``. Writing a 1 to a bit
clears that vector, and a 0 leaves the bit as it was.
* ``LEAF_EN_SET(i)`` and ``LEAF_EN_CLEAR(i)`` enable and disable the vectors
of leaf ``i``, one bit per vector.
* ``TOP_EN_SET`` and ``TOP_EN_CLEAR`` enable and disable subtrees, one bit per
subtree.
* ``LEAF_TRIGGER`` takes a vector number and latches that vector, exactly as
the vector's own source would. It is write-only. The self-test uses it.
nova-core does not read ``TOP``. "Servicing the tree" says why.
Every set and clear register acts per bit: a 1 performs the action for that
bit, and a 0 leaves the bit alone. No register needs a read-modify-write.
GIN delivers a vector to the CPU only when its leaf enable bit and its
subtree's TOP enable bit are both set. The enables do not affect the latch. The
source of a disabled vector still sets its ``LEAF`` bit. ``TOP`` does not show
that bit, so reading the leaf is the only way to see it.
How a unit interrupt reaches the CPU
------------------------------------
A unit does not write a ``LEAF`` register. Each unit has an interrupt control
register that GSP firmware programs. The control register holds the unit's
vector, the GFID that identifies the PCIe function whose tree receives the
interrupt, and one enable bit per destination: the CPU, the GSP, and the other
on-chip processors. When the unit has an event::
1. The unit sends GIN an interrupt message carrying the vector, the GFID,
and the destination enables from its control register.
2. In the tree of each destination that the message selects, GIN sets bit
(vector % 32) of LEAF(vector / 32).
3. If the vector and its subtree are enabled in the CPU tree, GIN raises
the PCI interrupt.
Because firmware assigns the vectors, nova-core does not hardcode which vector
belongs to which unit, with two exceptions. The hardware headers of every
supported chipset define the GSP event as vector 155, and GSP firmware's own
interrupt table uses that definition. The CPU doorbell is vector 129. Pre-Hopper
hardware fixes that number, and GSP firmware keeps it on Hopper and later.
nova-core names both by number. A driver can fetch the full unit-to-vector
table from the GSP by RPC, and nouveau does. nova-core does not, because a
fixed vector needs no lookup.
Edge-triggered delivery
-----------------------
A ``LEAF`` bit is a latch. Its source sets it on a rising edge, and it stays set
until the CPU clears it. A source that stays high does not set the bit again.
GIN raises the PCI interrupt for a subtree when the subtree's enabled pending
state goes from low to high::
Per vector, in leaf i at bit b:
LEAF(i)[b] AND LEAF_EN(i)[b]
Per subtree N, across leaves 2N and 2N + 1:
OR of every enabled pending bit -> TOP[N]
Delivery for subtree N:
TOP[N] AND TOP_EN[N] -> rising edge -> PCI interrupt
``TOP_EN`` applies after the summary, so disabling a subtree stops delivery
without changing what ``TOP`` reports.
Three consequences:
* Code that must find every pending vector reads the leaves. A vector that
latched while disabled is not in ``TOP``.
* Writing ``TOP_EN_SET`` for a subtree with an enabled pending bit produces a
new edge. GIN delivers an interrupt for a pending bit left uncleared as soon
as its subtree is enabled again.
* A source that holds its signal high produces no new edge after the CPU
clears the leaf bit. Such a source has to re-emit its interrupt. The falcons
do that through ``INTR_RETRIGGER`` (see "Retriggering a falcon").
Delivery over PCI
=================
GIN delivers the tree's interrupts to the CPU as MSI or MSI-X, whichever Linux
grants. nova-core requests MSI-X first and falls back to MSI, and never uses
INTx.
MSI has a single message, and every subtree raises that one message, so one
Linux IRQ serves the whole tree.
MSI-X gives each subtree its own table entry, at the index equal to the subtree
number. Linux masks every entry until a driver requests its Linux IRQ number,
and a masked entry sends no message: the GPU records the interrupt in the MSI-X
pending bit array, where it stays until Linux unmasks the entry. An entry that
the driver never requests is never unmasked. A driver that enables a subtree
without requesting that subtree's entry loses every interrupt from that
subtree, with nothing reported: the leaf and TOP registers show the vector
pending and enabled while no handler runs.
The serviced-subtree invariant
------------------------------
Every subtree enabled at ``TOP`` has an allocated PCI vector with a registered
handler.
MSI satisfies this with its single message. MSI-X needs one allocated entry per
serviced subtree, and a PCI allocation cannot be sparse, so nova-core requests
entries 0 through the highest serviced subtree::
MSI-X, with subtree 2 serviced:
subtree 0 -> entry 0 allocated, no handler, stays masked
subtree 1 -> entry 1 allocated, no handler, stays masked
subtree 2 -> entry 2 handler here, and its rearm covers subtree 2
MSI, with any serviced set:
every serviced subtree -> the one allocated PCI vector, whose
handler's rearm covers the whole serviced set
An allocated entry whose subtree nova-core does not service costs nothing. The
entry stays masked, and a disabled subtree raises no interrupt.
nova-core services one subtree. The GSP event, vector 155, is in leaf 4, which
is in subtree 2. OpenRM's headers place its UVM_SHARED interrupt category in
subtree 2 on every chipset that nova-core supports. The self-test doorbell,
vector 129, is in the same leaf, and the test allocates its own vectors for it.
Rearming PCI interrupt delivery
-------------------------------
A message-signaled interrupt is delivered once per edge, and the PCI side
delivers no further interrupt until the CPU rearms it. The rearm operation
depends on the GPU family and on the interrupt type that Linux granted:
================== ===== ===========================================
Architecture Type Rearm operation
================== ===== ===========================================
Turing through Ada MSI write the configuration-mirror EOI register
Hopper and later MSI clear then set the serviced TOP enables
Any MSI-X clear then set the handler's own TOP enable
================== ===== ===========================================
The end-of-interrupt register is ``NV_XVE_CYA_2`` in the BAR0 mirror of PCI
configuration space, and the value written does not matter. The ``TOP_EN``
cycle produces a new delivery edge. The MSI forms cover every serviced subtree,
because one message serves all of them. The MSI-X form covers one subtree,
because each serviced subtree has its own entry and its own handler.
A handler rearms once per delivered interrupt, on every path, including the
path where it finds its vector not pending. A handler that skips the rearm
receives no further interrupts.
OpenRM makes the same split. It writes the configuration-space EOI for MSI on
pre-Hopper chipsets, and cycles the TOP enables of the subtrees that it
services for Hopper-plus MSI and for MSI-X.
Servicing the tree
==================
Servicing a leaf has a required order: read its pending bits, then clear them.
Clearing a leaf before reading it discards every vector latched in it, and no
register reports the loss. In nova-core, reading a leaf produces the handle that
clears it, so that the wrong order does not compile. The handle clears exactly
the bits that it read, so a vector that latched after the read stays pending.
A handler clears its bit before it services the vector. Clearing afterwards
would discard an interrupt that the source raised while the handler ran.
nova-core services the tree in two ways.
The notification path services one vector. It reads the vector's leaf, clears
only the vector's bit, and rearms. The subtree stays enabled, and a vector
pending beside it in the same leaf keeps its bit set for the code that services
that vector. The GSP event handler takes this path, and so does the self-test
handler.
The startup drain walks the whole tree, because it must clear whatever is
pending across every subtree rather than one known vector. It disables the
serviced subtrees at ``TOP``, reads and clears every implemented leaf, and
leaves the subtrees disabled for its caller to enable once the caller is ready
for deliveries. The drain reads every leaf rather than descending from ``TOP``,
because sources latch vectors during boot while those vectors are disabled, and
``TOP`` does not show them. OpenRM's stall-interrupt path reads every leaf for
the same reason.
The two paths as register operations::
Startup drain, run once during probe:
write TOP_EN_CLEAR = serviced stop new deliveries
for each implemented leaf i:
pending = read LEAF(i)
write LEAF(i) = pending clear what was read
(returns with TOP_EN still clear)
Notification, the subtree stays enabled:
pending = read LEAF(leaf) is the handler's bit set?
write LEAF(leaf) = bit clear that one bit
rearm PCI interrupt delivery
The drain clears every pending bit, including bits that nova-core never
services. An uncleared bit holds its subtree in the pending state, and enabling
that subtree again would deliver an interrupt for a vector that no handler
services.
The drain's ``TOP_EN_CLEAR`` is not a rearm, and pre-Hopper MSI rearms through
the configuration mirror, which the drain never writes. An interrupt delivered
before probe had no handler to rearm it, so the tree reset rearms explicitly
after the drain. A TOP-enable rearm leaves the serviced subtrees enabled, so
the reset then disables them again.
nova-core does not serialize access to the tree. The GSP event handler
touches only its own leaf, and the drain runs during probe, before that handler
is registered.
Per-architecture differences
============================
The tree is the same on every supported GPU except for its size, which changes
at Hopper:
=================== ====== ======== ====================
GPUs Leaves Subtrees Implemented subtrees
=================== ====== ======== ====================
Turing, Ampere, Ada 8 4 ``0x0f``
Hopper, Blackwell 16 8 ``0xff``
=================== ====== ======== ====================
The interrupt HAL provides the leaf count, and the subtree count and the
implemented-subtree set derive from it. A subtree that the chipset does not
implement has no TOP bit, so building a tree that services one fails with
``EINVAL``. Vectors 129 and 155 are in the 8-leaf tree, so every supported
chipset has them.
OpenRM's headers assign every interrupt category of a 16-leaf tree to leaves 0
through 11. The drain reads all 16, because a vector can be latched in any
implemented leaf.
The HAL performs the rearm as well (see "Rearming PCI interrupt delivery"). Two
falcon properties also differ by family, and the falcon HAL carries them:
Turing falcons have no ``INTR_RETRIGGER``, and the RISC-V routing registers
moved at GA102 (see "Retriggering a falcon").
The GSP event
=============
When the GSP has output for the CPU, it writes messages into the GSP-to-CPU
queue in shared memory and raises SWGEN0, one of the software-generated
interrupt causes of the GSP falcon. SWGEN0 is routed to the host, at vector
155, leaf 4 bit 27, in subtree 2.
The queue carries notifications (log records, error records, lifecycle events)
and command replies. A thread waiting for a reply reads the queue itself, so
the interrupt is only the trigger to drain the queue (see "Draining the
GSP-to-CPU queue").
The falcon latches every cause that it raises, SWGEN0 among them, in its
``IRQSTAT`` register. The handler services the host-routed causes and clears
their latches, and then it writes ``INTR_RETRIGGER`` so that the falcon re-emits
any cause that latched in the meantime. "Retriggering a falcon" has the details.
Draining the queue takes the command-queue mutex and walks shared memory, so it
cannot run in hard interrupt context. nova-core registers a threaded handler,
under the name ``nova-core`` in ``/proc/interrupts``. The top half runs in hard
interrupt context and reads and writes only registers, and it wakes the IRQ
thread to drain the queue::
GSP writes messages into the GSP-to-CPU queue
GSP raises SWGEN0
GIN sets bit 27 of LEAF(4), and subtree 2 becomes pending
PCI interrupt -> Linux IRQ -> top half, in hard interrupt context:
read LEAF(4), and if bit 27 is clear, rearm and return
clear bit 27 (the subtree stays enabled)
read the falcon causes routed to the host, clearing SWGEN0 if set
for every other host cause: log it, clear its latch, and read the
host causes back
if the clear ended all of them: retrigger the falcon
otherwise: disable vector 155 at its leaf and skip the retrigger
rearm PCI interrupt delivery
wake the IRQ thread if SWGEN0 was set
IRQ thread, which may sleep:
take the command-queue mutex and drain the GSP-to-CPU queue
A halt and a posted message can be pending together, so the top half services
every cause that the status reports.
A drain fails when a message's framing or checksum is bad (see "Draining the
GSP-to-CPU queue"). The message stays at the queue head, so every later drain
would fail the same way, and the IRQ thread disables vector 155 and logs the
failure, which leaves the queue unserviced until the device is reset.
The handler, the self-test, and the rest of the driver read BAR0 through one
shared mapping. nova-core unregisters an interrupt handler when the device
unbinds, so a handler runs only while the mapping exists.
Retriggering a falcon
---------------------
A falcon signals the tree when its set of host-routed causes goes from empty to
non-empty. A cause left latched keeps the set non-empty, so no later cause
signals the tree, and the vector is lost. For a cause that stays latched, the
handler can clear the tree leaf first or the falcon latch first, and the loss
is the same.
``IRQSTAT`` latches every cause in the falcon, including the causes routed to
the falcon's own RISC-V core and owned by the firmware running on it. A host
handler owns only the causes that ``PRISCV_RISCV_IRQMASK`` and
``PRISCV_RISCV_IRQDEST`` both select, so it intersects ``IRQSTAT`` with both
before it reads or clears a cause. OpenRM computes the same intersection in
``kflcnRiscvReadIntrStatus``. GA100 keeps the Turing offsets of the two routing
registers and GA102 moves them, so the offsets change at GA102 rather than at
the Ampere boundary. The handler masks no cause: ``PRISCV_RISCV_IRQMASK`` is
read-only to the host, and ``FALCON_IRQMASK`` has no effect on host routing on
a RISC-V falcon.
``INTR_RETRIGGER`` makes the falcon re-emit its host-routed causes into the
tree, which supplies the transition that clearing the leaf lost. The handler
writes ``INTR_RETRIGGER`` only on a path where it ended every cause that it
read, because a re-emitted cause that stays set arrives again at once and
on every pass after that.
``IRQSCLR`` ends a latch and does not end the source behind it, so a cause
driven from outside the falcon stays set after the write. On Blackwell the
fault-containment and ECC causes are driven that way: they appear in
``IRQSTAT`` but come from ``PRISCV_RISCV_FAULT_CONTAINMENT_SRCSTAT`` and
``PGSP_ECC_INTR_STATUS``, and only a device reset ends them. So the handler
clears the latch of every host cause other than SWGEN0, reads the host causes
back, and retriggers only when the read-back is empty. When a cause is still
set, the handler disables vector 155 instead and reports that the device needs
a reset. Disabling loses no notification: the cause that is still set holds the
host-routed set non-empty, so the falcon would signal nothing further either
way. OpenRM makes the same choice, and ``kgspService_TU102`` skips
``kflcnIntrRetrigger`` once it has recorded a fatal error.
The handler cannot distinguish a fault cause that arrives after the clear from
one that the clear failed to end, so it disables the vector in that case too.
Both mean that the GSP has faulted.
Turing falcons have no ``INTR_RETRIGGER``, so a Turing handler cannot re-create
a transition that it has lost. It must leave no host cause latched: it reads the
host-routed status once and takes every cause that the status reports, rather
than stopping at the first one that it recognizes. One window stays open. A
cause that arrives after the handler has read the status is not in the value
that the handler clears, so it stays latched after the leaf has been cleared,
and no later cause from that falcon signals the tree. OpenRM has the same window
on Turing, where ``kflcnIntrRetrigger`` does nothing.
Enabling the GSP event
----------------------
SWGEN0 is a latch, and the GSP drives no new edge into the tree while it stays
set. nova-core's GSP boot code consumes the GSP's notifications by polling the
queue, which leaves the latch set and leaves pending bits in the tree. The
handoff from polling to interrupts has a required order::
disable every implemented vector drop enables left by boot, or by a
driver that ran before this one
drain the tree clear stale pending bits
rearm PCI interrupt delivery required under pre-Hopper MSI, which
no later step rearms
disable subtree 2 at TOP a TOP-enable rearm enabled it again
clear the SWGEN0 latch so the next message makes an edge
register the threaded handler no delivery can reach it yet
enable vector 155 at LEAF(4) the subtree is still disabled
enable subtree 2 at TOP deliveries become possible here
drain the GSP-to-CPU queue messages posted before the clear
The first four steps are the tree reset. nova-core runs them, and clears the
latch, before it registers the handler. Registering unmasks the PCI interrupt,
and GIN would then deliver a vector that boot left enabled to a handler that
services one vector and has no way to service any other. OpenRM clears every
leaf enable at the same point for the same reason.
nova-core clears the latch after the drain. If nova-core cleared the latch
first, a message posted before the drain could set it again, along with bit 27
of LEAF(4). The drain would then clear the leaf bit while the latch stays set,
and no later message would signal the tree. Clearing after the drain can
instead leave the leaf bit pending with the latch already clear. Enabling the
subtree then delivers one interrupt whose ``IRQSTAT`` reads zero. The top half
clears the leaf bit, rearms, and does not wake the IRQ thread, and the queue
drain that follows reads the message.
Clearing the latch makes the first interrupt possible. A message that the GSP
posted before that clear produces no interrupt, so the sequence ends by
draining the queue.
The subtree is enabled at ``TOP`` only once the handler is registered. A
TOP-enable rearm sets the enables that it cycles, so the reset disables the
serviced subtrees again after its rearm, and the tree reaches the registration
with subtree 2 disabled under every rearm method. On teardown nova-core disables
the vector at its leaf, so that no vector in the subtree can be delivered, then
calls ``free_irq()``, and disables the subtree last. Disabling the subtree
earlier would let a handler still in flight enable it again through the
``TOP_EN`` cycle of its rearm, which would leave the subtree enabled with no
handler registered. The driver's ``Gpu`` object, which owns the tree, holds the
subtree enable, and a handler's registration holds only the enable of its own
vector, so that tearing down one handler does not disable a subtree that another
handler shares. nova-core tears down the registration before it frees the queue
that the handler drains and the falcon that the handler reads, and before it
unloads the GSP.
Draining the GSP-to-CPU queue
-----------------------------
The queue carries command replies and unsolicited events, and a message's
function code says which it is.
* A function code that matches the awaited reply: the message is decoded and
returned to the caller that sent the command.
* Anything else is an event. An OS error record and a robust-channel record
are logged at error level, and an unrecognized function code at warning
level. The other known events (GSP logs, libos prints, assertion records,
lifecycle notices) need no action and get no line of their own, because the
receive trace at debug level already records every message's arrival with
its sequence number, function code, and length.
The sequence number takes no part in the match, because the GSP does not echo
the command's sequence number on every reply. On r570 the reply to
``UnloadingGuestDriver`` carries sequence 0.
The read pointer advances past every message that passes framing and checksum
validation, whether it matched or was an event. A matched message that is too
short to decode is the exception, and it stays at the queue head.
A message's length is inside the region that the checksum covers, so once the
framing or the checksum fails there is no trustworthy length with which to skip
the message. Such a message also stays at the queue head, and every later
receive fails on it with ``EIO`` until the device is reset.
The polling path and the IRQ thread both read the queue under the command-queue
mutex. Replies and events share one queue and one read pointer, so one lock is
held across the whole drain. A thread waiting for a reply logs each event that
arrives before the reply and keeps waiting. One deadline of 5 seconds applies to
the whole wait, rather than a fresh timeout after each message, and the thread
holds the mutex for the whole wait, so that no other caller consumes the message
that it waits for.
With one lock, a drain waits for an in-flight command's receive to finish or
time out. For log and error records that delay does not matter.
+128
View File
@@ -0,0 +1,128 @@
.. SPDX-License-Identifier: GPL-2.0
=========================
PRAMIN aperture mechanism
=========================
.. note::
The following description is approximate and current as of the Ampere
family. It may change for future generations and is intended to assist in
understanding the driver code.
Introduction
============
PRAMIN is a hardware aperture mechanism that provides CPU access to GPU Video
RAM (VRAM) before the GPU's Memory Management Unit (MMU) and page tables are
initialized. This 1 MiB sliding window, located at a fixed offset within BAR0,
is essential for setting up page tables and other critical GPU data structures
without relying on the GPU's MMU.
Architecture Overview
=====================
The PRAMIN aperture mechanism is logically implemented by the GPU's PBUS (PCIe
Bus Controller Unit) and provides a CPU-accessible window into VRAM through the
PCIe interface::
+-----------------+ PCIe +------------------------------+
| CPU |<----------->| GPU |
+-----------------+ | |
| +----------------------+ |
| | PBUS | |
| | (Bus Controller) | |
| | | |
| | +--------------+ <------------ [1]
| | | PRAMIN | | |
| | | Window | | |
| | | (1 MiB) | | |
| | +--------------+ | |
| | | | |
| +---------|------------+ |
| | |
| v |
| +----------------------+ <------- [2]
| | VRAM | |
| | (Several GiB) | |
| | | |
| | FB[0x0000000000] | |
| | ... | |
| | FB[0xFFFFFFFFFF] | |
| +----------------------+ |
+------------------------------+
[1] Window starts at BAR0 + 0x700000.
[2] Program PRAMIN to any 64 KiB-aligned VRAM boundary.
PBUS is responsible for, among other things, handling MMIO
accesses to the BAR registers.
PRAMIN Window Operation
=======================
The PRAMIN window provides a 1 MiB sliding aperture that can be repositioned
over the entire VRAM address space using the ``NV_PBUS_BAR0_WINDOW`` register.
Window Control Mechanism
-------------------------
::
NV_PBUS_BAR0_WINDOW Register (0x1700):
+-------+--------+--------------------------------------+
| 31:26 | 25:24 | 23:0 |
| RSVD | TARGET | BASE_ADDR |
| | | (bits 39:16 of VRAM address) |
+-------+--------+--------------------------------------+
The 24-bit BASE_ADDR field encodes bits [39:16] of the target VRAM address,
providing 40-bit (1 TiB) address space coverage with 64 KiB alignment.
TARGET field (bits 25:24):
- 0x0: VRAM (Video Memory)
- 0x1: Reserved (unused)
- 0x2: SYS_MEM_COH (Coherent System Memory)
- 0x3: SYS_MEM_NONCOH (Non-coherent System Memory)
.. note::
Nova only uses TARGET=VRAM (0x0) for video memory access. The SYS_MEM
target values are documented here for hardware completeness but are
not used by the driver.
64 KiB Alignment Requirement
----------------------------
The PRAMIN window must be aligned to 64 KiB boundaries in VRAM. This is enforced
by the ``BASE_ADDR`` field representing bits [39:16] of the target address::
VRAM Address Calculation:
actual_vram_addr = (BASE_ADDR << 16) + pramin_offset
Where:
- BASE_ADDR: 24-bit value from NV_PBUS_BAR0_WINDOW[23:0]
- pramin_offset: 20-bit offset within the PRAMIN window [0x00000-0xFFFFF]
Example Window Positioning:
+---------------------------------------------------------+
| VRAM Space |
| |
| 0x0000000000 +-----------------+ <-- 64 KiB aligned |
| | PRAMIN Window | |
| | (1 MiB) | |
| 0x00000FFFFF +-----------------+ |
| |
| | ^ |
| | | Window can slide |
| v | to any 64 KiB-aligned boundary |
| |
| 0x0123400000 +-----------------+ <-- 64 KiB aligned |
| | PRAMIN Window | |
| | (1 MiB) | |
| 0x01234FFFFF +-----------------+ |
| |
| ... |
| |
| 0xFFFFF00000 +-----------------+ <-- 64 KiB aligned |
| | PRAMIN Window | |
| | (1 MiB) | |
| 0xFFFFFFFFFF +-----------------+ |
+---------------------------------------------------------+
+1 -1
View File
@@ -33,7 +33,7 @@ A good example from nova-core would be the ``Chipset`` enum type, which defines
the value ``AD102``. When probing the GPU the value ``0x192`` can be read from a
certain register indication the chipset AD102. Hence, the enum value ``AD102``
should be derived from the number ``0x192``. Currently, nova-core uses a custom
implementation (``Chipset::from_u32`` for this.
implementation (``Chipset::from_u32``) for this.
Instead, it would be desirable to have something like the ``FromPrimitive``
trait [1] from the num crate.
+2
View File
@@ -34,3 +34,5 @@ vGPU manager VFIO driver and the nova-drm driver.
core/fwsec
core/falcon
core/tlv
core/pramin
core/interrupts
+21 -27
View File
@@ -29,21 +29,6 @@ refactorings already and are an expert in the specific area
Subsystem-wide refactorings
===========================
Open-code drm_simple_encoder_init()
-----------------------------------
The helper drm_simple_encoder_init() was supposed to simplify encoder
initialization. Instead it only added an intermediate layer between atomic
modesetting and the DRM driver.
The task here is to remove drm_simple_encoder_init(). Search for a driver
that calls drm_simple_encoder_init() and inline the helper. The driver will
also need its own instance of drm_encoder_funcs.
Contact: Thomas Zimmermann, respective driver maintainer
Level: Easy
Replace struct drm_simple_display_pipe with regular atomic helpers
------------------------------------------------------------------
@@ -499,21 +484,30 @@ Contact: Douglas Anderson <[email protected]>
Level: Starter
Remove devm_drm_put_bridge()
----------------------------
Remove panel_bridge API functions
---------------------------------
Due to how the panel bridge handles the drm_bridge object lifetime, special
care must be taken to dispose of the drm_bridge object when the
panel_bridge is removed. This is currently managed using
devm_drm_put_bridge(), but that is an unsafe, temporary workaround. To fix
that, the DRM panel lifetime needs to be reworked. After the rework is
done, remove devm_drm_put_bridge() and the TODO in
drm_panel_bridge_remove().
Now that a drm_bridge is embedded into every drm_panel, the panel_bridge is
not needed anymore. Its APIs are just accessing the embedded drm_bridge
transparently, and the are deprecated.
Contact: Maxime Ripard <[email protected]>,
Luca Ceresoli <[email protected]>
The funcions to remove are:
- drm_panel_bridge_add_typed()
- devm_drm_of_get_bridge()
- drmm_of_get_bridge()
- drm_panel_bridge_remove()
- drmm_drm_panel_bridge_release()
- devm_drm_panel_bridge_release()
- their direct or indirect callers
Level: Intermediate
Convert users of those functions and eventually remove them when unused, or
make them static drm_panel.c functions when used internally by the
``drm_panel.c`` code.
Contact: Luca Ceresoli <[email protected]>
Maxime Ripard <[email protected]>,
Level: Easy
Convert users of of_drm_find_bridge() to of_drm_find_and_get_bridge()
---------------------------------------------------------------------
+1
View File
@@ -35,3 +35,4 @@ The display, or :ref:`drm-kms`, support for drm/xe is provided by
xe-drm-usage-stats.rst
xe_configfs
xe_gt_stats
xe_sigid
+3
View File
@@ -6,3 +6,6 @@ Migrate Layer
.. kernel-doc:: drivers/gpu/drm/xe/xe_migrate_doc.h
:doc: Migrate Layer
.. kernel-doc:: drivers/gpu/drm/xe/xe_migrate.c
:doc: ULLS (Ultra Low Latency Submission) for migration jobs
+14
View File
@@ -0,0 +1,14 @@
.. SPDX-License-Identifier: (GPL-2.0+ OR MIT)
========
Xe SIGID
========
.. kernel-doc:: drivers/gpu/drm/xe/abi/xe_sigid_abi.h
:doc: Xe Error Signatures (SIGID)
Signature Identifiers
=====================
.. kernel-doc:: drivers/gpu/drm/xe/abi/xe_sigid_abi.h
:internal:
+80
View File
@@ -69,6 +69,37 @@ attribute-sets:
name: error-value
type: u32
doc: Current value of the requested error counter.
-
name: error-threshold
type: u32
doc: Error threshold of the counter.
-
name: error-event-attrs
attributes:
-
name: device-name
type: string
doc: Device (PCI BDF, UUID) that reported the error.
-
name: node-id
type: u32
doc: ID of the node that reported the error.
-
name: node-name
type: string
doc: Name of the node that reported the error.
-
name: error-id
type: u32
doc: ID of the error counter.
-
name: error-name
type: string
doc: Name of the error.
-
name: error-value
type: u32
doc: Current value of the error counter.
operations:
list:
@@ -124,3 +155,52 @@ operations:
do:
request:
attributes: *id-attrs
-
name: error-event
doc: >-
Report an error event to userspace.
The event includes the device, node and error information
of the error that triggered the event.
attribute-set: error-event-attrs
mcgrp: error-report
event:
attributes:
- device-name
- node-id
- node-name
- error-id
- error-name
- error-value
-
name: get-error-threshold
doc: >-
Retrieve error threshold of a given counter.
The response includes the id, the name, and current threshold
of the counter.
attribute-set: error-counter-attrs
flags: [admin-perm]
do:
request:
attributes: *id-attrs
reply:
attributes:
- error-id
- error-name
- error-threshold
-
name: set-error-threshold
doc: >-
Set error threshold of a given counter.
attribute-set: error-counter-attrs
flags: [admin-perm]
do:
request:
attributes:
- node-id
- error-id
- error-threshold
mcast-groups:
list:
-
name: error-report
+66 -8
View File
@@ -1228,7 +1228,7 @@ AMD KFD
M: Felix Kuehling <[email protected]>
L: [email protected]
S: Supported
T: git https://gitlab.freedesktop.org/agd5f/linux.git
T: git https://gitlab.freedesktop.org/drm/amdgpu/kernel.git
F: drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd*.[ch]
F: drivers/gpu/drm/amd/amdkfd/
F: drivers/gpu/drm/amd/include/cik_structs.h
@@ -2343,6 +2343,7 @@ ARM MALI TYR DRM DRIVER
M: Daniel Almeida <[email protected]>
M: Alice Ryhl <[email protected]>
L: [email protected]
L: [email protected]
S: Supported
W: https://rust-for-linux.com/tyr-gpu-driver
W: https://drm.pages.freedesktop.org/maintainer-tools/drm-rust.html
@@ -7736,6 +7737,7 @@ F: fs/dlm/
DMA BUFFER SHARING FRAMEWORK
M: Sumit Semwal <[email protected]>
M: Christian König <[email protected]>
M: Philipp Stanner <[email protected]>
L: [email protected]
L: [email protected]
L: [email protected] (moderated for non-subscribers)
@@ -7749,6 +7751,8 @@ F: include/linux/dma-buf.h
F: include/linux/dma-buf/
F: include/linux/dma-resv.h
F: rust/helpers/dma-resv.c
F: rust/helpers/dma_fence.c
F: rust/kernel/dma_buf/
K: \bdma_(?:buf|fence|resv)\b
DMA GENERIC OFFLOAD ENGINE SUBSYSTEM
@@ -8399,6 +8403,13 @@ T: git https://gitlab.freedesktop.org/drm/misc/kernel.git
F: Documentation/devicetree/bindings/display/panel/novatek,nt36523.yaml
F: drivers/gpu/drm/panel/panel-novatek-nt36523.c
DRM DRIVER FOR NOVATEK NT36532 PANELS
M: Junjie Cao <[email protected]>
S: Maintained
T: git https://gitlab.freedesktop.org/drm/misc/kernel.git
F: Documentation/devicetree/bindings/display/panel/novatek,nt36532.yaml
F: drivers/gpu/drm/panel/panel-novatek-nt36532.c
DRM DRIVER FOR NOVATEK NT36672A PANELS
M: Sumit Semwal <[email protected]>
S: Maintained
@@ -8412,6 +8423,13 @@ S: Maintained
F: Documentation/devicetree/bindings/display/panel/novatek,nt37801.yaml
F: drivers/gpu/drm/panel/panel-novatek-nt37801.c
DRM DRIVER FOR NOVATEK NT51021 PANELS
M: Ryan Brue <[email protected]>
M: Nickolay Goppen <[email protected]>
S: Maintained
F: Documentation/devicetree/bindings/display/panel/novatek,nt51021.yaml
F: drivers/gpu/drm/panel/panel-novatek-nt51021.c
DRM DRIVER FOR NVIDIA GEFORCE/QUADRO GPUS
M: Lyude Paul <[email protected]>
M: Danilo Krummrich <[email protected]>
@@ -8501,6 +8519,12 @@ S: Maintained
F: Documentation/devicetree/bindings/display/panel/raydium,rm67191.yaml
F: drivers/gpu/drm/panel/panel-raydium-rm67191.c
DRM DRIVER FOR RAYDIUM RM69220 PANELS
M: Alexander Koskovich <[email protected]>
S: Maintained
F: Documentation/devicetree/bindings/display/panel/raydium,rm69220.yaml
F: drivers/gpu/drm/panel/panel-raydium-rm69220.c
DRM DRIVER FOR SAMSUNG DB7430 PANELS
M: Linus Walleij <[email protected]>
S: Maintained
@@ -8756,7 +8780,9 @@ T: git https://gitlab.freedesktop.org/drm/rust/kernel.git
F: drivers/gpu/drm/nova/
F: drivers/gpu/drm/tyr/
F: drivers/gpu/nova-core/
F: rust/helpers/dma_fence.c
F: rust/helpers/gpu.c
F: rust/kernel/dma_buf/
F: rust/kernel/drm/
F: rust/kernel/gpu.rs
F: rust/kernel/gpu/
@@ -9128,6 +9154,21 @@ F: include/drm/drm_gpuvm.h
F: rust/helpers/drm_gpuvm.c
F: rust/kernel/drm/gpuvm/
DRM GPUSVM
M: Matthew Brost <[email protected]>
R: Thomas Hellström <[email protected]>
R: Himal Prasad Ghimiray <[email protected]>
R: Honglei Huang <[email protected]>
L: [email protected]
S: Supported
T: git https://gitlab.freedesktop.org/drm/misc/kernel.git
F: drivers/gpu/drm/drm_gpusvm.c
F: drivers/gpu/drm/drm_pagemap.c
F: drivers/gpu/drm/drm_pagemap_util.c
F: include/drm/drm_gpusvm.h
F: include/drm/drm_pagemap.h
F: include/drm/drm_pagemap_util.h
DRM LOG
M: Jocelyn Falempe <[email protected]>
M: Javier Martinez Canillas <[email protected]>
@@ -9155,9 +9196,12 @@ S: Supported
T: git https://gitlab.freedesktop.org/drm/misc/kernel.git
F: drivers/gpu/drm/drm_draw.c
F: drivers/gpu/drm/drm_draw_internal.h
F: drivers/gpu/drm/drm_panic*.c
F: drivers/gpu/drm/tests/drm_panic_test.c
F: include/drm/drm_panic*
F: drivers/gpu/drm/drm_panic.c
F: drivers/gpu/drm/drm_panic_helper.c
F: drivers/gpu/drm/drm_panic_internal.h
F: drivers/gpu/drm/tests/drm_panic_helper_test.c
F: include/drm/drm_panic.h
F: include/drm/drm_panic_helper.h
DRM PANIC QR CODE
M: Jocelyn Falempe <[email protected]>
@@ -9166,7 +9210,7 @@ L: [email protected]
L: [email protected]
S: Supported
T: git https://gitlab.freedesktop.org/drm/misc/kernel.git
F: drivers/gpu/drm/drm_panic_qr.rs
F: drivers/gpu/drm/drm_panic_helper_qr.rs
DRM PRIVACY-SCREEN CLASS
M: Hans de Goede <[email protected]>
@@ -9176,11 +9220,25 @@ T: git https://gitlab.freedesktop.org/drm/misc/kernel.git
F: drivers/gpu/drm/drm_privacy_screen*
F: include/drm/drm_privacy_screen*
DRM RAS
M: Riana Tauro <[email protected]>
M: Rodrigo Vivi <[email protected]>
R: Aravind Iddamsetty <[email protected]>
R: Raag Jadav <[email protected]>
L: [email protected]
S: Supported
T: git https://gitlab.freedesktop.org/drm/misc/kernel.git
F: Documentation/netlink/specs/drm_ras.yaml
F: Documentation/gpu/drm-ras.rst
F: drivers/gpu/drm/drm_ras*
F: include/drm/drm_ras*
DRM TTM SUBSYSTEM
M: Christian Koenig <[email protected]>
M: Huang Rui <ray.huang@amd.com>
M: Natalie Vock <[email protected]>
M: Arun Pravin <arunpravin.paneerselvam@amd.com>
R: Matthew Auld <[email protected]>
R: Matthew Brost <[email protected]>
R: Thomas Hellström <[email protected]>
L: [email protected]
S: Maintained
T: git https://gitlab.freedesktop.org/drm/misc/kernel.git
@@ -22842,7 +22900,7 @@ L: [email protected]
S: Supported
B: https://gitlab.freedesktop.org/drm/amd/-/issues
C: irc://irc.oftc.net/radeon
T: git https://gitlab.freedesktop.org/agd5f/linux.git
T: git https://gitlab.freedesktop.org/drm/amdgpu/kernel.git
F: Documentation/gpu/amdgpu/
F: drivers/gpu/drm/amd/
F: drivers/gpu/drm/ci/xfails/amd*
@@ -139,6 +139,7 @@
compatible = "brcm,bcm2835-v3d";
reg = <0x7ec00000 0x1000>;
interrupts = <1 10>;
resets = <&pm BCM2835_RESET_V3D>;
};
};
+39
View File
@@ -118,6 +118,45 @@ void amdxdna_vbnv_init(struct amdxdna_dev *xdna)
amdxdna_update_vbnv(xdna, info->rev_vbnv_tbl, rev);
}
void amdxdna_io_stats_job_start(struct amdxdna_client *client)
{
int depth;
guard(spinlock)(&client->io_stats.lock);
depth = client->io_stats.job_depth++;
if (!depth)
client->io_stats.start_time = ktime_get_ns();
}
void amdxdna_io_stats_job_done(struct amdxdna_client *client)
{
u64 busy_ns;
int depth;
guard(spinlock)(&client->io_stats.lock);
depth = --client->io_stats.job_depth;
if (!depth) {
busy_ns = ktime_get_ns() - client->io_stats.start_time;
client->io_stats.start_time = 0;
client->io_stats.busy_time += busy_ns;
}
}
u64 amdxdna_io_stats_busy_time_ns(struct amdxdna_client *client)
{
u64 busy_ns;
guard(spinlock)(&client->io_stats.lock);
busy_ns = client->io_stats.busy_time;
if (client->io_stats.job_depth)
busy_ns += ktime_get_ns() - client->io_stats.start_time;
return busy_ns;
}
int amdxdna_get_metadata(struct aie_device *aie,
struct amdxdna_client *client,
struct amdxdna_drm_get_info *args)
+4
View File
@@ -117,4 +117,8 @@ void aie_smu_fini(struct smu_device *smu);
int aie_smu_set_clocks(struct smu_device *smu, u32 *npuclk, u32 *hclk);
int aie_smu_set_dpm(struct smu_device *smu, u32 dpm_level);
void amdxdna_io_stats_job_start(struct amdxdna_client *client);
void amdxdna_io_stats_job_done(struct amdxdna_client *client);
u64 amdxdna_io_stats_busy_time_ns(struct amdxdna_client *client);
#endif /* _AIE_H_ */
+102 -30
View File
@@ -7,6 +7,7 @@
#include <drm/drm_device.h>
#include <drm/drm_gem.h>
#include <drm/drm_gem_shmem_helper.h>
#include <drm/drm_managed.h>
#include <drm/drm_print.h>
#include <drm/drm_syncobj.h>
#include <linux/hmm.h>
@@ -99,9 +100,9 @@ static void aie2_job_put(struct amdxdna_sched_job *job)
static void aie2_hwctx_stop(struct amdxdna_dev *xdna, struct amdxdna_hwctx *hwctx,
struct drm_sched_job *bad_job)
{
drm_sched_stop(&hwctx->priv->sched, bad_job);
drm_sched_stop(hwctx->priv->sched, bad_job);
aie2_destroy_context(xdna->dev_handle, hwctx);
drm_sched_start(&hwctx->priv->sched, 0);
drm_sched_start(hwctx->priv->sched, 0);
}
static int aie2_hwctx_restart(struct amdxdna_dev *xdna, struct amdxdna_hwctx *hwctx)
@@ -289,6 +290,7 @@ aie2_sched_resp_handler(void *handle, void __iomem *data, size_t size)
int ret = 0;
u32 status;
amdxdna_io_stats_job_done(job->hwctx->client);
cmd_abo = job->cmd_bo;
if (unlikely(job->job_timeout)) {
@@ -321,6 +323,8 @@ aie2_sched_drvcmd_resp_handler(void *handle, void __iomem *data, size_t size)
struct amdxdna_sched_job *job = handle;
int ret = 0;
amdxdna_io_stats_job_done(job->hwctx->client);
if (unlikely(!data || size != sizeof(u32))) {
job->drv_cmd->result = U32_MAX;
ret = -EINVAL;
@@ -343,6 +347,7 @@ aie2_sched_cmdlist_resp_handler(void *handle, void __iomem *data, size_t size)
u32 cmd_status;
int ret = 0;
amdxdna_io_stats_job_done(job->hwctx->client);
cmd_abo = job->cmd_bo;
if (unlikely(job->job_timeout)) {
@@ -404,6 +409,7 @@ aie2_sched_job_run(struct drm_sched_job *sched_job)
kref_get(&job->refcnt);
fence = dma_fence_get(job->fence);
amdxdna_io_stats_job_start(job->hwctx->client);
if (job->drv_cmd) {
switch (job->drv_cmd->opcode) {
case SYNC_DEBUG_BO:
@@ -431,6 +437,7 @@ aie2_sched_job_run(struct drm_sched_job *sched_job)
out:
if (ret) {
amdxdna_io_stats_job_done(job->hwctx->client);
dma_fence_put(job->fence);
aie2_job_put(job);
mmput(job->mm);
@@ -659,10 +666,19 @@ static void aie2_ctx_syncobj_destroy(struct amdxdna_hwctx *hwctx)
drm_syncobj_put(hwctx->priv->syncobj);
}
int aie2_hwctx_init(struct amdxdna_hwctx *hwctx)
void aie2_hwctx_sched_fini(struct amdxdna_dev_hdl *ndev)
{
struct amdxdna_client *client = hwctx->client;
struct amdxdna_dev *xdna = client->xdna;
int i;
for (i = 0; i < ndev->priv->hwctx_limit; i++)
drm_sched_fini(&ndev->hwctx_sched[i]);
ida_destroy(&ndev->hwctx_sched_ida);
}
int aie2_hwctx_sched_init(struct amdxdna_dev_hdl *ndev)
{
struct amdxdna_dev *xdna = ndev->aie.xdna;
const struct drm_sched_init_args args = {
.ops = &sched_ops,
.num_rqs = DRM_SCHED_PRIORITY_COUNT,
@@ -673,7 +689,55 @@ int aie2_hwctx_init(struct amdxdna_hwctx *hwctx)
.name = "amdxdna_js",
.dev = xdna->ddev.dev,
};
struct drm_gpu_scheduler *sched;
int i, ret;
ndev->hwctx_sched = drmm_kcalloc(&xdna->ddev, ndev->priv->hwctx_limit,
sizeof(*ndev->hwctx_sched), GFP_KERNEL);
if (!ndev->hwctx_sched)
return -ENOMEM;
for (i = 0; i < ndev->priv->hwctx_limit; i++) {
ret = drm_sched_init(&ndev->hwctx_sched[i], &args);
if (ret) {
XDNA_ERR(xdna, "Failed to init DRM scheduler. ret %d", ret);
goto fini_sched;
}
}
ida_init(&ndev->hwctx_sched_ida);
return 0;
fini_sched:
while (--i >= 0)
drm_sched_fini(&ndev->hwctx_sched[i]);
return ret;
}
static struct drm_gpu_scheduler *aie2_hwctx_sched_alloc(struct amdxdna_dev_hdl *ndev)
{
int ret;
ret = ida_alloc_range(&ndev->hwctx_sched_ida, 0,
ndev->priv->hwctx_limit - 1, GFP_KERNEL);
if (ret < 0)
return ERR_PTR(ret);
return &ndev->hwctx_sched[ret];
}
static void aie2_hwctx_sched_free(struct amdxdna_dev_hdl *ndev,
struct drm_gpu_scheduler *sched)
{
ida_free(&ndev->hwctx_sched_ida, sched - ndev->hwctx_sched);
}
int aie2_hwctx_init(struct amdxdna_hwctx *hwctx)
{
struct amdxdna_client *client = hwctx->client;
struct amdxdna_dev *xdna = client->xdna;
struct amdxdna_dev_hdl *ndev = xdna->dev_handle;
struct amdxdna_hwctx_priv *priv;
struct amdxdna_gem_obj *heap;
int i, ret;
@@ -683,13 +747,26 @@ int aie2_hwctx_init(struct amdxdna_hwctx *hwctx)
return -ENOMEM;
hwctx->priv = priv;
priv->sched = aie2_hwctx_sched_alloc(ndev);
if (IS_ERR(priv->sched)) {
ret = PTR_ERR(priv->sched);
goto free_priv;
}
ret = drm_sched_entity_init(&priv->entity, DRM_SCHED_PRIORITY_NORMAL,
&priv->sched, 1, NULL);
if (ret) {
XDNA_ERR(xdna, "Failed to initial sched entity. ret %d", ret);
goto free_sched;
}
mutex_lock(&client->mm_lock);
heap = xa_load(&client->dev_heap_xa, 0);
if (!heap) {
XDNA_ERR(xdna, "The client dev heap object not exist");
mutex_unlock(&client->mm_lock);
ret = -ENOENT;
goto free_priv;
goto free_entity;
}
drm_gem_object_get(to_gobj(heap));
mutex_unlock(&client->mm_lock);
@@ -722,30 +799,16 @@ int aie2_hwctx_init(struct amdxdna_hwctx *hwctx)
priv->cmd_buf[i] = abo;
}
sched = &priv->sched;
mutex_init(&priv->io_lock);
fs_reclaim_acquire(GFP_KERNEL);
might_lock(&priv->io_lock);
fs_reclaim_release(GFP_KERNEL);
ret = drm_sched_init(sched, &args);
if (ret) {
XDNA_ERR(xdna, "Failed to init DRM scheduler. ret %d", ret);
goto free_cmd_bufs;
}
ret = drm_sched_entity_init(&priv->entity, DRM_SCHED_PRIORITY_NORMAL,
&sched, 1, NULL);
if (ret) {
XDNA_ERR(xdna, "Failed to initial sched entiry. ret %d", ret);
goto free_sched;
}
ret = aie2_hwctx_col_list(hwctx);
if (ret) {
XDNA_ERR(xdna, "Create col list failed, ret %d", ret);
goto free_entity;
goto free_cmd_bufs;
}
ret = amdxdna_pm_resume_get_locked(xdna);
@@ -791,19 +854,20 @@ suspend_put:
amdxdna_pm_suspend_put(xdna);
free_col_list:
kfree(hwctx->col_list);
free_entity:
drm_sched_entity_destroy(&priv->entity);
free_sched:
drm_sched_fini(&priv->sched);
free_cmd_bufs:
for (i = 0; i < ARRAY_SIZE(priv->cmd_buf); i++) {
if (!priv->cmd_buf[i])
continue;
amdxdna_gem_heap_free(client, priv->cmd_buf[i]);
drm_gem_object_put(to_gobj(priv->cmd_buf[i]));
}
amdxdna_gem_unpin(heap);
put_heap:
drm_gem_object_put(to_gobj(heap));
free_entity:
drm_sched_entity_destroy(&priv->entity);
free_sched:
aie2_hwctx_sched_free(ndev, priv->sched);
free_priv:
kfree(priv);
return ret;
@@ -811,18 +875,20 @@ free_priv:
void aie2_hwctx_fini(struct amdxdna_hwctx *hwctx)
{
struct amdxdna_dev_hdl *ndev;
struct amdxdna_dev *xdna;
int idx;
xdna = hwctx->client->xdna;
ndev = xdna->dev_handle;
XDNA_DBG(xdna, "%s sequence number %lld", hwctx->name, hwctx->priv->seq);
aie2_hwctx_wait_for_idle(hwctx);
/* Request fw to destroy hwctx and cancel the rest pending requests */
drm_sched_stop(&hwctx->priv->sched, NULL);
drm_sched_stop(hwctx->priv->sched, NULL);
aie2_release_resource(hwctx);
drm_sched_start(&hwctx->priv->sched, 0);
drm_sched_start(hwctx->priv->sched, 0);
mutex_unlock(&xdna->dev_lock);
drm_sched_entity_destroy(&hwctx->priv->entity);
@@ -833,11 +899,17 @@ void aie2_hwctx_fini(struct amdxdna_hwctx *hwctx)
atomic64_read(&hwctx->job_free_cnt));
mutex_lock(&xdna->dev_lock);
drm_sched_fini(&hwctx->priv->sched);
aie2_hwctx_sched_free(ndev, hwctx->priv->sched);
aie2_ctx_syncobj_destroy(hwctx);
for (idx = 0; idx < ARRAY_SIZE(hwctx->priv->cmd_buf); idx++)
for (idx = 0; idx < ARRAY_SIZE(hwctx->priv->cmd_buf); idx++) {
/*
* The open/close will never be called for driver allocated
* dev bo. Call amdxdna_gem_heap_free explicitly.
*/
amdxdna_gem_heap_free(hwctx->client, hwctx->priv->cmd_buf[idx]);
drm_gem_object_put(to_gobj(hwctx->priv->cmd_buf[idx]));
}
amdxdna_gem_unpin(hwctx->priv->heap);
drm_gem_object_put(to_gobj(hwctx->priv->heap));
+4 -5
View File
@@ -343,15 +343,14 @@ void aie2_error_async_events_free(struct amdxdna_dev_hdl *ndev)
kfree(events);
}
int aie2_error_async_events_alloc(struct amdxdna_dev_hdl *ndev)
int aie2_error_async_events_alloc(struct amdxdna_dev_hdl *ndev, u32 num_events)
{
struct amdxdna_dev *xdna = ndev->aie.xdna;
u32 total_col = ndev->total_col;
u32 total_size = ASYNC_BUF_SIZE * total_col;
u32 total_size = ASYNC_BUF_SIZE * num_events;
struct async_events *events;
int i, ret;
events = kzalloc_flex(*events, event, total_col);
events = kzalloc_flex(*events, event, num_events);
if (!events)
return -ENOMEM;
@@ -361,7 +360,7 @@ int aie2_error_async_events_alloc(struct amdxdna_dev_hdl *ndev)
goto free_events;
}
events->size = total_size;
events->event_cnt = total_col;
events->event_cnt = num_events;
events->wq = alloc_ordered_workqueue("async_wq", 0);
if (!events->wq) {
+11 -8
View File
@@ -261,8 +261,9 @@ int aie2_create_context(struct amdxdna_dev_hdl *ndev, struct amdxdna_hwctx *hwct
goto del_ctx_req;
}
/* +1 for the config_cu message that can be in-flight concurrently */
ret = xdna_mailbox_start_channel(hwctx->priv->mbox_chann, &x2i, &i2x,
intr_reg, ret);
intr_reg, ret, HWCTX_MAX_CMDS + 1);
if (ret) {
XDNA_ERR(xdna, "Not able to create channel");
ret = -EINVAL;
@@ -277,6 +278,7 @@ int aie2_create_context(struct amdxdna_dev_hdl *ndev, struct amdxdna_hwctx *hwct
free_channel:
xdna_mailbox_free_channel(hwctx->priv->mbox_chann);
hwctx->priv->mbox_chann = NULL;
del_ctx_req:
aie2_destroy_context_req(ndev, hwctx->fw_ctx_id);
return ret;
@@ -485,7 +487,7 @@ int aie2_register_asyn_event_msg(struct amdxdna_dev_hdl *ndev, dma_addr_t addr,
req.buf_size = size;
XDNA_DBG(ndev->aie.xdna, "Register addr 0x%llx size 0x%x", addr, size);
return xdna_mailbox_send_msg(ndev->aie.mgmt_chann, &msg, TX_TIMEOUT);
return xdna_mailbox_send_msg(ndev->aie.mgmt_chann, &msg);
}
int aie2_config_cu(struct amdxdna_hwctx *hwctx,
@@ -544,7 +546,8 @@ int aie2_config_cu(struct amdxdna_hwctx *hwctx,
msg.handle = hwctx;
msg.opcode = MSG_OP_CONFIG_CU;
msg.notify_cb = notify_cb;
return xdna_mailbox_send_msg(chann, &msg, TX_TIMEOUT);
return xdna_mailbox_send_msg(chann, &msg);
}
static int aie2_init_exec_cu_req(struct amdxdna_gem_obj *cmd_bo, void *req,
@@ -949,7 +952,7 @@ int aie2_execbuf(struct amdxdna_hwctx *hwctx, struct amdxdna_sched_job *job,
print_hex_dump_debug("cmd: ", DUMP_PREFIX_OFFSET, 16, 4, &req,
0x40, false);
ret = xdna_mailbox_send_msg(chann, &msg, TX_TIMEOUT);
ret = xdna_mailbox_send_msg(chann, &msg);
if (ret) {
XDNA_ERR(xdna, "Send message failed");
return ret;
@@ -1038,7 +1041,7 @@ int aie2_cmdlist_multi_execbuf(struct amdxdna_hwctx *hwctx,
msg.send_size = sizeof(req);
print_hex_dump_debug("cmdlist msg: ", DUMP_PREFIX_OFFSET, 16, 4,
&req, msg.send_size, false);
ret = xdna_mailbox_send_msg(chann, &msg, TX_TIMEOUT);
ret = xdna_mailbox_send_msg(chann, &msg);
if (ret) {
XDNA_ERR(xdna, "Send message failed");
return ret;
@@ -1085,7 +1088,7 @@ int aie2_cmdlist_single_execbuf(struct amdxdna_hwctx *hwctx,
msg.send_size = sizeof(req);
print_hex_dump_debug("cmdlist msg: ", DUMP_PREFIX_OFFSET, 16, 4,
&req, msg.send_size, false);
ret = xdna_mailbox_send_msg(chann, &msg, TX_TIMEOUT);
ret = xdna_mailbox_send_msg(chann, &msg);
if (ret) {
XDNA_ERR(hwctx->client->xdna, "Send message failed");
return ret;
@@ -1121,7 +1124,7 @@ int aie2_sync_bo(struct amdxdna_hwctx *hwctx, struct amdxdna_sched_job *job,
msg.send_size = sizeof(req);
msg.opcode = MSG_OP_SYNC_BO;
ret = xdna_mailbox_send_msg(chann, &msg, TX_TIMEOUT);
ret = xdna_mailbox_send_msg(chann, &msg);
if (ret) {
XDNA_ERR(xdna, "Send message failed");
return ret;
@@ -1156,7 +1159,7 @@ int aie2_config_debug_bo(struct amdxdna_hwctx *hwctx, struct amdxdna_sched_job *
msg.send_size = sizeof(req);
msg.opcode = MSG_OP_CONFIG_DEBUG_BO;
return xdna_mailbox_send_msg(chann, &msg, TX_TIMEOUT);
return xdna_mailbox_send_msg(chann, &msg);
}
int aie2_query_app_health(struct amdxdna_dev_hdl *ndev, u32 context_id,
+1
View File
@@ -530,6 +530,7 @@ enum aie2_dev_revision {
AIE2_DEV_REVISION_GPT1,
AIE2_DEV_REVISION_GPT2,
AIE2_DEV_REVISION_GPT3,
AIE2_DEV_REVISION_GRGH,
AIE2_DEV_REVISION_UNKN,
};
+25 -5
View File
@@ -24,6 +24,7 @@
#include "aie2_pci.h"
#include "aie2_solver.h"
#include "amdxdna_ctx.h"
#include "amdxdna_error.h"
#include "amdxdna_gem.h"
#include "amdxdna_mailbox.h"
#include "amdxdna_pci_drv.h"
@@ -385,11 +386,21 @@ static int aie2_hw_start(struct amdxdna_dev *xdna)
}
xdna_mailbox_intr_reg = ndev->aie.mgmt_i2x.mb_head_ptr_reg + 4;
/*
* At most AMDXDNA_MAX_ASYNC_EVENT_BUFS async event messages plus one
* management command can be unresponded. The async slots stay busy
* until firmware reports an error. Every management command holds
* dev_lock across aie_send_mgmt_msg_wait(), and the response clears
* the slot before that wait returns, so a second ioctl blocks on
* dev_lock rather than observing -ENOBUFS. Async re-registration
* takes the same lock after the RX path has freed its slot.
*/
ret = xdna_mailbox_start_channel(ndev->aie.mgmt_chann,
&ndev->aie.mgmt_x2i,
&ndev->aie.mgmt_i2x,
xdna_mailbox_intr_reg,
mgmt_mb_irq);
mgmt_mb_irq,
AMDXDNA_MAX_ASYNC_EVENT_BUFS + 1);
if (ret) {
XDNA_ERR(xdna, "failed to start management mailbox channel");
ret = -EINVAL;
@@ -414,7 +425,7 @@ static int aie2_hw_start(struct amdxdna_dev *xdna)
goto stop_fw;
}
ret = aie2_error_async_events_alloc(ndev);
ret = aie2_error_async_events_alloc(ndev, AMDXDNA_MAX_ASYNC_EVENT_BUFS);
if (ret) {
XDNA_ERR(xdna, "Allocate async events failed, ret %d", ret);
goto stop_fw;
@@ -503,10 +514,16 @@ static int aie2_init(struct amdxdna_dev *xdna)
ndev->priv = xdna->dev_info->dev_priv;
ndev->aie.xdna = xdna;
ret = aie2_hwctx_sched_init(ndev);
if (ret)
return ret;
for (i = 0; i < ARRAY_SIZE(npu_fw); i++) {
fw_full_path = kasprintf(GFP_KERNEL, "%s%s", ndev->priv->fw_path, npu_fw[i]);
if (!fw_full_path)
return -ENOMEM;
if (!fw_full_path) {
ret = -ENOMEM;
goto free_hwctx_sched;
}
ret = firmware_request_nowarn(&fw, fw_full_path, &pdev->dev);
kfree(fw_full_path);
@@ -519,7 +536,7 @@ static int aie2_init(struct amdxdna_dev *xdna)
if (ret) {
XDNA_ERR(xdna, "failed to request_firmware %s, ret %d",
ndev->priv->fw_path, ret);
return ret;
goto free_hwctx_sched;
}
ret = pcim_enable_device(pdev);
@@ -623,6 +640,8 @@ stop_hw:
aie2_hw_stop(xdna);
release_fw:
release_firmware(fw);
free_hwctx_sched:
aie2_hwctx_sched_fini(ndev);
return ret;
}
@@ -631,6 +650,7 @@ static void aie2_fini(struct amdxdna_dev *xdna)
{
amdxdna_pm_fini(xdna);
aie2_hw_stop(xdna);
aie2_hwctx_sched_fini(xdna->dev_handle);
}
static int aie2_get_aie_status(struct amdxdna_client *client,
+6 -2
View File
@@ -107,7 +107,7 @@ struct amdxdna_hwctx_priv {
struct amdxdna_gem_obj *heap;
void *mbox_chann;
struct drm_gpu_scheduler sched;
struct drm_gpu_scheduler *sched;
struct drm_sched_entity entity;
struct mutex io_lock; /* protect seq and cmd order */
@@ -152,6 +152,8 @@ struct amdxdna_dev_hdl {
u32 total_col;
struct amdxdna_drm_query_aie_version version;
struct aie2_exec_msg_ops *exec_msg_ops;
struct drm_gpu_scheduler *hwctx_sched;
struct ida hwctx_sched_ida;
/* power management and clock*/
enum amdxdna_power_mode_type pw_mode;
@@ -242,7 +244,7 @@ int aie2_pm_set_mode(struct amdxdna_dev_hdl *ndev, enum amdxdna_power_mode_type
int aie2_pm_set_dpm(struct amdxdna_dev_hdl *ndev, u32 dpm_level);
/* aie2_error.c */
int aie2_error_async_events_alloc(struct amdxdna_dev_hdl *ndev);
int aie2_error_async_events_alloc(struct amdxdna_dev_hdl *ndev, u32 num_events);
void aie2_error_async_events_free(struct amdxdna_dev_hdl *ndev);
int aie2_error_async_msg_thread(void *data);
int aie2_get_array_async_error(struct amdxdna_dev_hdl *ndev,
@@ -294,6 +296,8 @@ int aie2_update_prop_time_quota(struct amdxdna_dev_hdl *ndev, u32 us);
/* aie2_hwctx.c */
int aie2_hwctx_init(struct amdxdna_hwctx *hwctx);
void aie2_hwctx_fini(struct amdxdna_hwctx *hwctx);
int aie2_hwctx_sched_init(struct amdxdna_dev_hdl *ndev);
void aie2_hwctx_sched_fini(struct amdxdna_dev_hdl *ndev);
int aie2_hwctx_config(struct amdxdna_hwctx *hwctx, u32 type, u64 value, void *buf, u32 size);
int aie2_hwctx_sync_debug_bo(struct amdxdna_hwctx *hwctx, u32 debug_bo_hdl);
void aie2_hwctx_suspend(struct amdxdna_client *client);
+7 -1
View File
@@ -12,6 +12,7 @@
#include "aie.h"
#include "aie4_msg_priv.h"
#include "aie4_pci.h"
#include "amdxdna_error.h"
#include "amdxdna_mailbox.h"
#include "amdxdna_mailbox_helper.h"
#include "amdxdna_pci_drv.h"
@@ -181,11 +182,16 @@ static int aie4_mailbox_start(struct amdxdna_dev *xdna,
goto free_channel;
}
/*
* At any given time, at most AMDXDNA_MAX_ASYNC_EVENT_BUFS async event
* messages plus 1 other management command can be unresponded.
*/
ret = xdna_mailbox_start_channel(ndev->aie.mgmt_chann,
&ndev->aie.mgmt_x2i,
&ndev->aie.mgmt_i2x,
NO_IOHUB,
mgmt_mb_irq);
mgmt_mb_irq,
AMDXDNA_MAX_ASYNC_EVENT_BUFS + 1);
if (ret) {
XDNA_ERR(xdna, "failed to start management mailbox channel");
ret = -EINVAL;
+10
View File
@@ -56,4 +56,14 @@ enum amdxdna_error_module {
(FIELD_PREP(AMDXDNA_EXTRA_ERR_COL_MASK, col) | \
FIELD_PREP(AMDXDNA_EXTRA_ERR_ROW_MASK, row))
/*
* Maximum number of async event buffers. This value is used in two places:
* (1) to size the management mailbox channel, which must happen before
* total_col is known from firmware; (2) to allocate the async event pool,
* which runs after total_col is known. Both counts must match. The channel
* is sized AMDXDNA_MAX_ASYNC_EVENT_BUFS + 1 (one extra for other management
* commands), so the pool cannot exceed AMDXDNA_MAX_ASYNC_EVENT_BUFS slots.
*/
#define AMDXDNA_MAX_ASYNC_EVENT_BUFS 4
#endif /* _AMDXDNA_ERROR_H_ */
+323 -142
View File
@@ -14,6 +14,7 @@
#include <linux/dma-direct.h>
#include <linux/iosys-map.h>
#include <linux/pagemap.h>
#include <linux/swap.h>
#include <linux/vmalloc.h>
#include "amdxdna_cbuf.h"
@@ -37,6 +38,7 @@ amdxdna_init_dev_bo(struct amdxdna_gem_obj *dev_bo)
struct amdxdna_gem_obj *heap;
u64 heap_addr, exp_heap_uva;
u32 heap_id;
int ret;
if (xa_empty(&client->dev_heap_xa)) {
XDNA_DBG(xdna, "Empty heap xa");
@@ -58,24 +60,32 @@ amdxdna_init_dev_bo(struct amdxdna_gem_obj *dev_bo)
heap = xa_load(&client->dev_heap_xa, heap_id);
exp_heap_uva = amdxdna_gem_uva(heap);
heap_addr = amdxdna_gem_dev_addr(heap);
dev_bo->heap_start_id = heap_id;
dev_bo->mem.uva = dev_bo->mm_node.start - heap_addr + exp_heap_uva;
for (; heap_id < client->dev_heap_nid; heap_id++) {
heap = xa_load(&client->dev_heap_xa, heap_id);
if (!heap) {
XDNA_ERR(xdna, "Failed to load heap %d", heap_id);
return -EINVAL;
ret = -EINVAL;
goto cleanup_heap_xa;
}
heap_addr = amdxdna_gem_uva(heap);
if (heap_addr == AMDXDNA_INVALID_ADDR) {
XDNA_ERR(xdna, "Heap %d is not mapped", heap_id);
return -EAGAIN;
ret = -EAGAIN;
goto cleanup_heap_xa;
}
if (heap_addr != exp_heap_uva) {
XDNA_ERR(xdna, "Heap %d uva is not contiguous", heap_id);
return -EINVAL;
ret = -EINVAL;
goto cleanup_heap_xa;
}
ret = xa_insert(&dev_bo->heap_xa, heap_id, heap, GFP_KERNEL);
if (ret) {
ret = -ENOMEM;
goto cleanup_heap_xa;
}
if (heap->dev_addr + heap->mem.size >=
@@ -87,12 +97,15 @@ amdxdna_init_dev_bo(struct amdxdna_gem_obj *dev_bo)
if (heap_id == client->dev_heap_nid) {
XDNA_DBG(xdna, "Can not find heap end");
return -EAGAIN;
ret = -EAGAIN;
goto cleanup_heap_xa;
}
dev_bo->heap_end_id = heap_id;
return 0;
cleanup_heap_xa:
xa_destroy(&dev_bo->heap_xa);
return ret;
}
static int
@@ -132,8 +145,7 @@ amdxdna_gem_heap_alloc(struct amdxdna_gem_obj *abo)
}
client->heap_usage += mem->size;
xa_for_each_range(&client->dev_heap_xa, heap_id, heap,
abo->heap_start_id, abo->heap_end_id)
xa_for_each(&abo->heap_xa, heap_id, heap)
drm_gem_object_get(to_gobj(heap));
unlock_out:
@@ -142,48 +154,16 @@ unlock_out:
return ret;
}
static void
amdxdna_gem_heap_free(struct amdxdna_gem_obj *abo)
void amdxdna_gem_heap_free(struct amdxdna_client *client, struct amdxdna_gem_obj *abo)
{
struct amdxdna_client *client = abo->client;
struct amdxdna_gem_obj *heap;
unsigned long heap_id;
mutex_lock(&client->mm_lock);
drm_mm_remove_node(&abo->mm_node);
client->heap_usage -= abo->mem.size;
xa_for_each_range(&client->dev_heap_xa, heap_id, heap,
abo->heap_start_id, abo->heap_end_id)
drm_gem_object_put(to_gobj(heap));
mutex_unlock(&client->mm_lock);
}
static struct amdxdna_gem_obj *
amdxdna_gem_create_obj(struct drm_device *dev, size_t size)
{
struct amdxdna_gem_obj *abo;
abo = kzalloc_obj(*abo);
if (!abo)
return ERR_PTR(-ENOMEM);
abo->pinned = false;
abo->assigned_hwctx = AMDXDNA_INVALID_CTX_HANDLE;
mutex_init(&abo->lock);
abo->mem.dma_addr = AMDXDNA_INVALID_ADDR;
abo->mem.uva = AMDXDNA_INVALID_ADDR;
abo->mem.size = size;
abo->open_ref = 0;
abo->internal = false;
INIT_LIST_HEAD(&abo->mem.umap_list);
return abo;
}
static void
amdxdna_gem_destroy_obj(struct amdxdna_gem_obj *abo)
{
@@ -253,13 +233,16 @@ static bool amdxdna_hmm_invalidate(struct mmu_interval_notifier *mni,
struct amdxdna_gem_obj *abo = mapp->abo;
struct amdxdna_dev *xdna;
if (!mmu_notifier_range_blockable(range))
return false;
if (mapp->unmapped)
return true;
xdna = to_xdna_dev(to_gobj(abo)->dev);
XDNA_DBG(xdna, "Invalidating range 0x%lx, 0x%lx, type %d",
mapp->range.start, mapp->range.end, abo->type);
if (!mmu_notifier_range_blockable(range))
return false;
down_write(&xdna->notifier_lock);
abo->mem.map_invalid = true;
mapp->invalid = true;
@@ -271,10 +254,8 @@ static bool amdxdna_hmm_invalidate(struct mmu_interval_notifier *mni,
if (range->event == MMU_NOTIFY_UNMAP) {
down_write(&xdna->notifier_lock);
if (!mapp->unmapped) {
queue_work(xdna->notifier_wq, &mapp->hmm_unreg_work);
mapp->unmapped = true;
}
mapp->unmapped = true;
queue_work(xdna->notifier_wq, &abo->hmm_unreg_work);
up_write(&xdna->notifier_lock);
}
@@ -301,33 +282,42 @@ static void amdxdna_hmm_unregister(struct amdxdna_gem_obj *abo,
down_write(&xdna->notifier_lock);
list_for_each_entry(mapp, &abo->mem.umap_list, node) {
if (!vma || compare_range(mapp, vma->vm_mm, vma->vm_start, vma->vm_end)) {
if (!mapp->unmapped) {
queue_work(xdna->notifier_wq, &mapp->hmm_unreg_work);
mapp->unmapped = true;
}
if (vma)
break;
}
if (!compare_range(mapp, vma->vm_mm, vma->vm_start, vma->vm_end))
continue;
mapp->unmapped = true;
queue_work(xdna->notifier_wq, &abo->hmm_unreg_work);
}
up_write(&xdna->notifier_lock);
}
static void amdxdna_hmm_unreg_umaps(struct amdxdna_gem_obj *abo, bool force)
{
struct amdxdna_dev *xdna = to_xdna_dev(to_gobj(abo)->dev);
struct amdxdna_umap *mapp, *tmp;
LIST_HEAD(dead);
down_write(&xdna->notifier_lock);
list_for_each_entry_safe(mapp, tmp, &abo->mem.umap_list, node) {
if (!force && !mapp->unmapped)
continue;
mapp->unmapped = true;
list_move(&mapp->node, &dead);
}
if (list_empty(&abo->mem.umap_list))
abo->mem.uva = AMDXDNA_INVALID_ADDR;
up_write(&xdna->notifier_lock);
list_for_each_entry_safe(mapp, tmp, &dead, node)
amdxdna_umap_put(mapp);
}
static void amdxdna_umap_release(struct kref *ref)
{
struct amdxdna_umap *mapp = container_of(ref, struct amdxdna_umap, refcnt);
struct amdxdna_gem_obj *abo = mapp->abo;
struct amdxdna_dev *xdna;
mmu_interval_notifier_remove(&mapp->notifier);
xdna = to_xdna_dev(to_gobj(mapp->abo)->dev);
down_write(&xdna->notifier_lock);
list_del(&mapp->node);
if (list_empty(&abo->mem.umap_list))
abo->mem.uva = AMDXDNA_INVALID_ADDR;
up_write(&xdna->notifier_lock);
kvfree(mapp->range.hmm_pfns);
kfree(mapp);
}
@@ -339,10 +329,10 @@ void amdxdna_umap_put(struct amdxdna_umap *mapp)
static void amdxdna_hmm_unreg_work(struct work_struct *work)
{
struct amdxdna_umap *mapp = container_of(work, struct amdxdna_umap,
hmm_unreg_work);
struct amdxdna_gem_obj *abo = container_of(work, struct amdxdna_gem_obj,
hmm_unreg_work);
amdxdna_umap_put(mapp);
amdxdna_hmm_unreg_umaps(abo, false);
}
static int amdxdna_hmm_register(struct amdxdna_gem_obj *abo,
@@ -387,6 +377,21 @@ static int amdxdna_hmm_register(struct amdxdna_gem_obj *abo,
goto free_map;
}
mapp->range.notifier = &mapp->notifier;
mapp->range.start = vma->vm_start;
mapp->range.end = vma->vm_end;
/*
* Access permissions are fixed at mmap() time. Changing them later
* with mprotect() is not supported: the range keeps requesting the
* original permissions, so the application may see a fault failure
* or an IOMMU fault.
*/
mapp->range.default_flags = HMM_PFN_REQ_FAULT;
if (vma->vm_flags & VM_WRITE)
mapp->range.default_flags |= HMM_PFN_REQ_WRITE;
mapp->abo = abo;
kref_init(&mapp->refcnt);
ret = mmu_interval_notifier_insert_locked(&mapp->notifier,
current->mm,
addr,
@@ -397,15 +402,6 @@ static int amdxdna_hmm_register(struct amdxdna_gem_obj *abo,
goto free_pfns;
}
mapp->range.notifier = &mapp->notifier;
mapp->range.start = vma->vm_start;
mapp->range.end = vma->vm_end;
mapp->range.default_flags = HMM_PFN_REQ_FAULT;
mapp->abo = abo;
kref_init(&mapp->refcnt);
INIT_WORK(&mapp->hmm_unreg_work, amdxdna_hmm_unreg_work);
down_write(&xdna->notifier_lock);
if (list_empty(&abo->mem.umap_list))
abo->mem.uva = addr;
@@ -421,17 +417,47 @@ free_map:
return ret;
}
static struct amdxdna_gem_obj *
amdxdna_gem_create_obj(struct drm_device *dev, size_t size)
{
struct amdxdna_gem_obj *abo;
abo = kzalloc_obj(*abo);
if (!abo)
return ERR_PTR(-ENOMEM);
abo->pinned = false;
abo->assigned_hwctx = AMDXDNA_INVALID_CTX_HANDLE;
mutex_init(&abo->lock);
abo->mem.dma_addr = AMDXDNA_INVALID_ADDR;
abo->mem.uva = AMDXDNA_INVALID_ADDR;
abo->mem.size = size;
abo->open_ref = 0;
abo->internal = false;
INIT_LIST_HEAD(&abo->mem.umap_list);
xa_init_flags(&abo->heap_xa, XA_FLAGS_ALLOC);
INIT_WORK(&abo->hmm_unreg_work, amdxdna_hmm_unreg_work);
return abo;
}
static void amdxdna_gem_dev_obj_free(struct drm_gem_object *gobj)
{
struct amdxdna_dev *xdna = to_xdna_dev(gobj->dev);
struct amdxdna_gem_obj *abo = to_xdna_obj(gobj);
struct amdxdna_gem_obj *heap;
unsigned long heap_id;
XDNA_DBG(xdna, "BO type %d xdna_addr 0x%llx", abo->type, amdxdna_gem_dev_addr(abo));
if (abo->pinned)
amdxdna_gem_unpin(abo);
amdxdna_gem_vunmap(abo);
amdxdna_gem_heap_free(abo);
xa_for_each(&abo->heap_xa, heap_id, heap)
drm_gem_object_put(to_gobj(heap));
xa_destroy(&abo->heap_xa);
drm_gem_object_release(gobj);
amdxdna_gem_destroy_obj(abo);
}
@@ -458,16 +484,16 @@ static int amdxdna_insert_pages(struct amdxdna_gem_obj *abo,
{
struct amdxdna_dev *xdna = to_xdna_dev(to_gobj(abo)->dev);
unsigned long num_pages = vma_pages(vma);
unsigned long offset = 0;
int ret;
if (!is_import_bo(abo)) {
ret = drm_gem_shmem_mmap(&abo->base, vma);
if (ret) {
XDNA_ERR(xdna, "Failed shmem mmap %d", ret);
return ret;
}
} else {
/*
* Until today there is not any use case to mmap with non-zero
* offset. Put an explicit check here.
*/
if (vma->vm_pgoff - drm_vma_node_start(&to_gobj(abo)->vma_node))
return -EINVAL;
if (is_import_bo(abo)) {
vma->vm_private_data = NULL;
vma->vm_ops = NULL;
ret = dma_buf_mmap(abo->dma_buf, vma, 0);
@@ -476,23 +502,28 @@ static int amdxdna_insert_pages(struct amdxdna_gem_obj *abo,
return ret;
}
amdxdna_mark_mapp_invalid(abo, vma);
/* Drop the reference drm_gem_mmap_obj() acquired.*/
drm_gem_object_put(to_gobj(abo));
return 0;
}
do {
vm_fault_t fault_ret;
ret = drm_gem_shmem_mmap(&abo->base, vma);
if (ret) {
XDNA_ERR(xdna, "Failed shmem mmap %d", ret);
return ret;
}
fault_ret = handle_mm_fault(vma, vma->vm_start + offset,
FAULT_FLAG_WRITE, NULL);
if (fault_ret & VM_FAULT_ERROR) {
XDNA_ERR(xdna, "Fault in page failed");
amdxdna_mark_mapp_invalid(abo, vma);
break;
}
offset += PAGE_SIZE;
} while (--num_pages);
vm_flags_mod(vma, VM_MIXEDMAP, VM_PFNMAP);
ret = vm_insert_pages(vma, vma->vm_start, abo->base.pages, &num_pages);
if (ret) {
XDNA_ERR(xdna, "Failed to insert pages %d", ret);
dma_resv_lock(to_gobj(abo)->resv, NULL);
drm_gem_shmem_put_pages_locked(&abo->base);
dma_resv_unlock(to_gobj(abo)->resv);
return ret;
}
return 0;
}
@@ -504,6 +535,10 @@ static int amdxdna_gem_obj_mmap(struct drm_gem_object *gobj,
struct amdxdna_gem_obj *abo = to_xdna_obj(gobj);
int ret;
XDNA_DBG(xdna, "BO map_offset 0x%llx type %d userptr 0x%lx size 0x%lx",
drm_vma_node_offset_addr(&gobj->vma_node), abo->type,
vma->vm_start, gobj->size);
ret = amdxdna_hmm_register(abo, vma);
if (ret)
return ret;
@@ -514,9 +549,6 @@ static int amdxdna_gem_obj_mmap(struct drm_gem_object *gobj,
goto hmm_unreg;
}
XDNA_DBG(xdna, "BO map_offset 0x%llx type %d userptr 0x%lx size 0x%lx",
drm_vma_node_offset_addr(&gobj->vma_node), abo->type,
vma->vm_start, gobj->size);
return 0;
hmm_unreg:
@@ -524,14 +556,99 @@ hmm_unreg:
return ret;
}
/*
* VM operations for amdxdna shmem VMAs that use VM_MIXEDMAP.
*
* drm_gem_shmem_vm_ops cannot be used on VM_MIXEDMAP VMAs because its fault
* handler calls vmf_insert_pfn() → vmf_insert_pfn_prot() which contains:
* BUG_ON((vma->vm_flags & VM_MIXEDMAP) && pfn_valid(pfn))
* All amdxdna shmem pages are ordinary struct pages so pfn_valid() is always
* true, making the combination fatal.
*
* These ops use vmf_insert_page() (struct-page based) instead, which is the
* correct API for VM_MIXEDMAP VMAs backed by real struct pages. The open and
* close handlers replicate drm_gem_shmem_vm_open/close using only exported
* symbols.
*/
static vm_fault_t amdxdna_gem_mixedmap_fault(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
struct drm_gem_object *gobj = vma->vm_private_data;
struct drm_gem_shmem_object *shmem = to_drm_gem_shmem_obj(gobj);
loff_t num_pages = gobj->size >> PAGE_SHIFT;
vm_fault_t ret = VM_FAULT_SIGBUS;
pgoff_t page_offset;
struct page *page;
/*
* Partial free of vma is unexpected. Otherwise, the wrong page
* will be faulted in and the user application may crash itself.
*/
page_offset = linear_page_delta(vma, vmf->address);
dma_resv_lock(gobj->resv, NULL);
if (!shmem->pages || shmem->madv < 0 || page_offset >= num_pages)
goto out;
page = shmem->pages[page_offset];
if (WARN_ON_ONCE(!page))
goto out;
/*
* Use vmf_insert_page() (struct-page path) not vmf_insert_pfn()
* (PFN path) because this VMA carries VM_MIXEDMAP.
*/
ret = vmf_insert_page(vma, vmf->address, page);
if (ret == VM_FAULT_NOPAGE)
folio_mark_accessed(page_folio(page));
out:
dma_resv_unlock(gobj->resv);
return ret;
}
static void amdxdna_gem_mixedmap_vm_open(struct vm_area_struct *vma)
{
struct drm_gem_object *gobj = vma->vm_private_data;
struct drm_gem_shmem_object *shmem = to_drm_gem_shmem_obj(gobj);
/*
* Bump pages_use_count so the page array stays alive for the new
* mapping copy created by fork(). Mirrors drm_gem_shmem_vm_open().
*/
dma_resv_lock(gobj->resv, NULL);
drm_WARN_ON_ONCE(gobj->dev, !refcount_inc_not_zero(&shmem->pages_use_count));
dma_resv_unlock(gobj->resv);
drm_gem_vm_open(vma);
}
static void amdxdna_gem_mixedmap_vm_close(struct vm_area_struct *vma)
{
struct drm_gem_object *gobj = vma->vm_private_data;
struct drm_gem_shmem_object *shmem = to_drm_gem_shmem_obj(gobj);
dma_resv_lock(gobj->resv, NULL);
drm_gem_shmem_put_pages_locked(shmem);
dma_resv_unlock(gobj->resv);
drm_gem_vm_close(vma);
}
static const struct vm_operations_struct amdxdna_gem_mixedmap_vm_ops = {
.fault = amdxdna_gem_mixedmap_fault,
.open = amdxdna_gem_mixedmap_vm_open,
.close = amdxdna_gem_mixedmap_vm_close,
};
static int amdxdna_gem_dmabuf_mmap(struct dma_buf *dma_buf, struct vm_area_struct *vma)
{
struct drm_gem_object *gobj = dma_buf->priv;
struct amdxdna_gem_obj *abo = to_xdna_obj(gobj);
unsigned long num_pages = vma_pages(vma);
int ret;
vma->vm_ops = &drm_gem_shmem_vm_ops;
vma->vm_ops = &amdxdna_gem_mixedmap_vm_ops;
vma->vm_private_data = gobj;
drm_gem_object_get(gobj);
@@ -541,16 +658,9 @@ static int amdxdna_gem_dmabuf_mmap(struct dma_buf *dma_buf, struct vm_area_struc
/* The buffer is based on memory pages. Fix the flag. */
vm_flags_mod(vma, VM_MIXEDMAP, VM_PFNMAP);
ret = vm_insert_pages(vma, vma->vm_start, abo->base.pages,
&num_pages);
if (ret)
goto close_vma;
return 0;
close_vma:
vma->vm_ops->close(vma);
return ret;
put_obj:
drm_gem_object_put(gobj);
return ret;
@@ -609,10 +719,8 @@ amdxdna_gem_skip_bo_usage(struct amdxdna_gem_obj *abo)
}
static void
amdxdna_gem_add_bo_usage(struct amdxdna_gem_obj *abo)
amdxdna_gem_add_bo_usage(struct amdxdna_client *client, struct amdxdna_gem_obj *abo)
{
struct amdxdna_client *client = abo->client;
if (amdxdna_gem_skip_bo_usage(abo))
return;
@@ -624,10 +732,8 @@ amdxdna_gem_add_bo_usage(struct amdxdna_gem_obj *abo)
}
static void
amdxdna_gem_del_bo_usage(struct amdxdna_gem_obj *abo)
amdxdna_gem_del_bo_usage(struct amdxdna_client *client, struct amdxdna_gem_obj *abo)
{
struct amdxdna_client *client = abo->client;
if (amdxdna_gem_skip_bo_usage(abo))
return;
@@ -643,8 +749,9 @@ static void amdxdna_gem_obj_free(struct drm_gem_object *gobj)
struct amdxdna_dev *xdna = to_xdna_dev(gobj->dev);
struct amdxdna_gem_obj *abo = to_xdna_obj(gobj);
amdxdna_hmm_unregister(abo, NULL);
flush_workqueue(xdna->notifier_wq);
/* No notifier survives this, so no new work can be queued. */
amdxdna_hmm_unreg_umaps(abo, true);
cancel_work_sync(&abo->hmm_unreg_work);
if (abo->pinned)
amdxdna_gem_unpin(abo);
@@ -663,12 +770,20 @@ static int amdxdna_gem_obj_open(struct drm_gem_object *gobj, struct drm_file *fi
{
struct amdxdna_dev *xdna = to_xdna_dev(gobj->dev);
struct amdxdna_gem_obj *abo = to_xdna_obj(gobj);
struct amdxdna_client *client;
int ret;
guard(mutex)(&abo->lock);
mutex_lock(&abo->lock);
if (abo->open_ref > 0 && filp->driver_priv != abo->client) {
mutex_unlock(&abo->lock);
return -EPERM;
}
abo->open_ref++;
if (abo->open_ref > 1)
if (abo->open_ref > 1) {
mutex_unlock(&abo->lock);
return 0;
}
/* Attached to the client when first opened by it. */
abo->client = filp->driver_priv;
@@ -679,26 +794,34 @@ static int amdxdna_gem_obj_open(struct drm_gem_object *gobj, struct drm_file *fi
if (ret) {
abo->open_ref--;
abo->client = NULL;
mutex_unlock(&abo->lock);
return ret;
}
}
client = abo->client;
mutex_unlock(&abo->lock);
amdxdna_gem_add_bo_usage(abo);
amdxdna_gem_add_bo_usage(client, abo);
return 0;
}
static void amdxdna_gem_obj_close(struct drm_gem_object *gobj, struct drm_file *filp)
{
struct amdxdna_gem_obj *abo = to_xdna_obj(gobj);
struct amdxdna_client *client = NULL;
guard(mutex)(&abo->lock);
mutex_lock(&abo->lock);
abo->open_ref--;
if (abo->open_ref == 0) {
amdxdna_gem_del_bo_usage(abo);
/* Detach from the client when last closed by it. */
client = abo->client;
abo->client = NULL;
}
mutex_unlock(&abo->lock);
if (client)
amdxdna_gem_del_bo_usage(client, abo);
}
static int amdxdna_gem_obj_vmap(struct drm_gem_object *obj, struct iosys_map *map)
@@ -734,19 +857,72 @@ static void amdxdna_gem_obj_vunmap(struct drm_gem_object *obj, struct iosys_map
drm_gem_shmem_object_vunmap(obj, map);
}
static int amdxdna_gem_dev_obj_open(struct drm_gem_object *gobj, struct drm_file *filp)
{
struct amdxdna_gem_obj *abo = to_xdna_obj(gobj);
guard(mutex)(&abo->lock);
if (filp->driver_priv != abo->client)
return -EPERM;
abo->open_ref++;
return 0;
}
static void amdxdna_gem_dev_obj_close(struct drm_gem_object *gobj, struct drm_file *filp)
{
struct amdxdna_gem_obj *abo = to_xdna_obj(gobj);
struct amdxdna_client *client = NULL;
mutex_lock(&abo->lock);
abo->open_ref--;
/*
* Freeing the heap allocation here, when the handle is closed, is
* intentional. DEV BOs are carved out of a per-client drm_mm heap;
* any subsequent allocation that lands on the same device address will
* also belong to the same client. If the user closes the handle while
* a job is still in flight the only consequence is self-inflicted
* corruption within their own context -- it cannot affect other
* processes. The GEM reference held by the in-flight job keeps the
* amdxdna_gem_obj struct alive until the job completes; it does not
* prevent the device address from being reclaimed by the allocator.
*
* Cross-process handle creation for DEV BOs is rejected in
* amdxdna_gem_dev_obj_open(), which prevents the following UAF:
* if a second process shared the handle via GEM flink and the
* original creator exited (freeing client), the importer would later
* reach open_ref == 0 here and call amdxdna_gem_heap_free() with a
* dangling abo->client pointer. Because cross-process opens are
* rejected, the process arriving here is always the owning client,
* which is still alive. abo->client is nulled out afterwards so that
* any code path running on a lingering GEM reference (e.g. an
* in-flight job) cannot silently dereference a stale pointer.
*/
if (abo->open_ref == 0) {
client = abo->client;
abo->client = NULL;
}
mutex_unlock(&abo->lock);
if (client)
amdxdna_gem_heap_free(client, abo);
}
static int amdxdna_gem_dev_obj_vmap(struct drm_gem_object *obj, struct iosys_map *map)
{
struct amdxdna_gem_obj *abo = to_xdna_obj(obj);
struct amdxdna_gem_obj *heap;
unsigned long index = 0;
void *base;
u64 offset;
/* vmap dev bo which is across more than 1 heap is not allowed */
if (abo->heap_start_id != abo->heap_end_id)
heap = xa_find(&abo->heap_xa, &index, ULONG_MAX, XA_PRESENT);
if (!heap)
return -ENOMEM;
heap = xa_load(&abo->client->dev_heap_xa, abo->heap_start_id);
if (!heap)
/* vmap dev bo which is across more than 1 heap is not allowed */
if (xa_find_after(&abo->heap_xa, &index, ULONG_MAX, XA_PRESENT))
return -ENOMEM;
base = amdxdna_gem_vmap(heap);
@@ -765,6 +941,8 @@ static struct dma_buf *amdxdna_gem_dev_obj_export(struct drm_gem_object *gobj, i
static const struct drm_gem_object_funcs amdxdna_gem_dev_obj_funcs = {
.free = amdxdna_gem_dev_obj_free,
.open = amdxdna_gem_dev_obj_open,
.close = amdxdna_gem_dev_obj_close,
.vmap = amdxdna_gem_dev_obj_vmap,
.export = amdxdna_gem_dev_obj_export,
};
@@ -780,7 +958,7 @@ static const struct drm_gem_object_funcs amdxdna_gem_shmem_funcs = {
.vmap = amdxdna_gem_obj_vmap,
.vunmap = amdxdna_gem_obj_vunmap,
.mmap = amdxdna_gem_obj_mmap,
.vm_ops = &drm_gem_shmem_vm_ops,
.vm_ops = &amdxdna_gem_mixedmap_vm_ops,
.export = amdxdna_gem_prime_export,
};
@@ -962,6 +1140,11 @@ amdxdna_drm_create_dev_heap_bo(struct drm_device *dev,
struct amdxdna_gem_obj *abo;
int ret;
if (!xdna->dev_info->dev_mem_size) {
XDNA_DBG(xdna, "Device heap is not supported");
return ERR_PTR(-EOPNOTSUPP);
}
WARN_ON(!is_power_of_2(xdna->dev_info->dev_mem_size));
XDNA_DBG(xdna, "Requested dev heap size 0x%llx", args->size);
if (!args->size || !IS_ALIGNED(args->size, xdna->dev_info->dev_mem_size)) {
@@ -1103,6 +1286,8 @@ int amdxdna_drm_create_bo_ioctl(struct drm_device *dev, void *data, struct drm_f
args->handle, args->type, amdxdna_gem_uva(abo),
amdxdna_gem_dev_addr(abo), abo->mem.size);
put_obj:
if (ret && abo->type == AMDXDNA_BO_DEV)
amdxdna_gem_heap_free(client, abo);
/* Dereference object reference. Handle holds it now. */
drm_gem_object_put(to_gobj(abo));
return ret;
@@ -1136,7 +1321,6 @@ static void amdxdna_bo_unpin(struct amdxdna_gem_obj *abo)
int amdxdna_gem_pin_nolock(struct amdxdna_gem_obj *abo)
{
struct amdxdna_client *client = abo->client;
struct amdxdna_gem_obj *heap;
unsigned long heap_id, last = ULONG_MAX;
int ret = 0;
@@ -1144,17 +1328,15 @@ int amdxdna_gem_pin_nolock(struct amdxdna_gem_obj *abo)
if (abo->type != AMDXDNA_BO_DEV)
return amdxdna_bo_pin(abo);
xa_for_each_range(&client->dev_heap_xa, heap_id, heap,
abo->heap_start_id, abo->heap_end_id) {
xa_for_each(&abo->heap_xa, heap_id, heap) {
ret = amdxdna_bo_pin(heap);
if (ret)
break;
last = heap_id;
}
if (ret && last <= abo->heap_end_id) {
xa_for_each_range(&client->dev_heap_xa, heap_id, heap,
abo->heap_start_id, last)
if (ret && last != ULONG_MAX) {
xa_for_each_range(&abo->heap_xa, heap_id, heap, 0, last)
amdxdna_bo_unpin(heap);
}
@@ -1179,8 +1361,7 @@ void amdxdna_gem_unpin(struct amdxdna_gem_obj *abo)
struct amdxdna_gem_obj *heap;
unsigned long heap_id;
xa_for_each_range(&abo->client->dev_heap_xa, heap_id, heap,
abo->heap_start_id, abo->heap_end_id)
xa_for_each(&abo->heap_xa, heap_id, heap)
amdxdna_bo_unpin(heap);
} else {
amdxdna_bo_unpin(abo);
@@ -1300,8 +1481,7 @@ int amdxdna_drm_sync_bo_ioctl(struct drm_device *dev,
u64 flush_start = bo_start + args->offset;
u64 flush_end = flush_start + args->size;
xa_for_each_range(&client->dev_heap_xa, heap_id, heap,
abo->heap_start_id, abo->heap_end_id) {
xa_for_each(&abo->heap_xa, heap_id, heap) {
u64 heap_start = amdxdna_gem_dev_addr(heap);
u64 heap_end = heap_start + heap->mem.size;
u64 start = max(flush_start, heap_start);
@@ -1328,7 +1508,7 @@ int amdxdna_drm_sync_bo_ioctl(struct drm_device *dev,
amdxdna_gem_unpin(abo);
if (ret) {
drm_WARN(&xdna->ddev, 1, "Can not get flush memory");
XDNA_DBG(xdna, "Flush BO %d failed, ret %d", args->handle, ret);
goto put_obj;
}
}
@@ -1336,7 +1516,8 @@ int amdxdna_drm_sync_bo_ioctl(struct drm_device *dev,
XDNA_DBG(xdna, "Sync bo %d offset 0x%llx, size 0x%llx\n",
args->handle, args->offset, args->size);
if (args->direction == SYNC_DIRECT_FROM_DEVICE)
if (abo->assigned_hwctx != AMDXDNA_INVALID_CTX_HANDLE &&
args->direction == SYNC_DIRECT_FROM_DEVICE)
ret = amdxdna_hwctx_sync_debug_bo(client, args->handle);
put_obj:
+3 -3
View File
@@ -14,7 +14,6 @@
struct amdxdna_umap {
struct mmu_interval_notifier notifier;
struct hmm_range range;
struct work_struct hmm_unreg_work;
struct amdxdna_gem_obj *abo;
struct list_head node;
struct kref refcnt;
@@ -43,11 +42,11 @@ struct amdxdna_gem_obj {
struct mutex lock; /* Protects: pinned, mem.kva, open_ref */
struct amdxdna_mem mem;
int open_ref;
struct work_struct hmm_unreg_work;
/* Below members are initialized when needed */
struct drm_mm_node mm_node; /* For AMDXDNA_BO_DEV */
u32 heap_start_id;
u32 heap_end_id;
struct xarray heap_xa;
u64 dev_addr; /* For heap bo */
u32 assigned_hwctx;
struct dma_buf *dma_buf;
@@ -104,6 +103,7 @@ static inline u64 amdxdna_obj_dma_addr(struct amdxdna_gem_obj *abo)
}
void amdxdna_umap_put(struct amdxdna_umap *mapp);
void amdxdna_gem_heap_free(struct amdxdna_client *client, struct amdxdna_gem_obj *abo);
struct drm_gem_object *
amdxdna_gem_create_shmem_object_cb(struct drm_device *dev, size_t size);
+150 -71
View File
@@ -9,7 +9,6 @@
#include <linux/interrupt.h>
#include <linux/iopoll.h>
#include <linux/slab.h>
#include <linux/xarray.h>
#define CREATE_TRACE_POINTS
#include <trace/events/amdxdna.h>
@@ -35,9 +34,12 @@
(_chann)->msix_irq, ##args); \
})
#define MSG_BUF_SZ(chann) \
(mailbox_get_ringbuf_size(chann, CHAN_RES_X2I) + \
sizeof(struct mailbox_msg))
#define MAGIC_VAL 0x1D000000U
#define MAGIC_VAL_MASK 0xFF000000
#define MAX_MSG_ID_ENTRIES 256
#define MSG_RX_TIMER 200 /* milliseconds */
#define MAILBOX_NAME "xdna_mailbox"
@@ -57,8 +59,6 @@ struct mailbox_channel {
struct xdna_mailbox_chann_res res[CHAN_RES_NUM];
int msix_irq;
u32 iohub_int_addr;
struct xarray chan_xa;
u32 next_msgid;
u32 x2i_tail;
/* Received msg related fields */
@@ -66,6 +66,12 @@ struct mailbox_channel {
struct work_struct rx_work;
u32 i2x_head;
bool bad_state;
/* Pre-allocated message buffer pool */
void *msg_buf;
u32 msg_buf_num;
u32 next_slot; /* next slot index to try for allocation */
struct mutex lock;
};
#define MSG_BODY_SZ GENMASK(10, 0)
@@ -90,12 +96,36 @@ struct mailbox_pkg {
#define TOMBSTONE 0xDEADFACE
struct mailbox_msg {
bool busy;
void *handle;
int (*notify_cb)(void *handle, void __iomem *data, size_t size);
size_t pkg_size; /* package size in bytes */
struct mailbox_pkg pkg;
};
/*
* Clear busy before invoking the callback so that a new submission can
* reuse the slot immediately - by the time notify_cb returns, the caller
* may already have submitted a new job that needs this slot.
*
* notify_cb and handle are cached in locals first. smp_store_release()
* ensures those loads complete before busy is cleared - without it the
* compiler or CPU could sink the loads past the store, letting a
* concurrent sender overwrite notify_cb/handle before we read them.
* Paired with smp_load_acquire() in mailbox_acquire_msgid().
*/
static int mailbox_msg_done(struct mailbox_msg *mb_msg, void __iomem *data, size_t size)
{
int (*notify_cb)(void *, void __iomem *, size_t) = mb_msg->notify_cb;
void *handle = mb_msg->handle;
/* Pairs with smp_load_acquire() in mailbox_acquire_msgid(). */
smp_store_release(&mb_msg->busy, false);
if (notify_cb)
return notify_cb(handle, data, size);
return 0;
}
static void mailbox_reg_write(struct mailbox_channel *mb_chann, u32 mbox_reg, u32 data)
{
struct xdna_mailbox_res *mb_res = &mb_chann->mb->res;
@@ -161,38 +191,57 @@ static inline int mailbox_validate_msgid(int msg_id)
return (msg_id & MAGIC_VAL_MASK) == MAGIC_VAL;
}
static int mailbox_acquire_msgid(struct mailbox_channel *mb_chann, struct mailbox_msg *mb_msg)
static struct mailbox_msg *mailbox_msg_ptr(struct mailbox_channel *mb_chann, u32 slot)
{
u32 msg_id;
int ret;
return mb_chann->msg_buf + slot * MSG_BUF_SZ(mb_chann);
}
ret = xa_alloc_cyclic_irq(&mb_chann->chan_xa, &msg_id, mb_msg,
XA_LIMIT(0, MAX_MSG_ID_ENTRIES - 1),
&mb_chann->next_msgid, GFP_NOWAIT);
if (ret < 0)
return ret;
static int mailbox_acquire_msgid(struct mailbox_channel *mb_chann)
{
struct mailbox_msg *mb_msg;
u32 slot;
u32 i;
/*
* Add MAGIC_VAL to the higher bits.
* msg_id = slot | MAGIC_VAL. Scan slots starting from next_slot.
* smp_load_acquire() pairs with smp_store_release() in mailbox_msg_done()
* to ensure that once we observe busy == false, notify_cb and handle are
* also visible as their post-completion values before we overwrite them.
*/
msg_id |= MAGIC_VAL;
return msg_id;
for (i = 0; i < mb_chann->msg_buf_num; i++) {
slot = (mb_chann->next_slot + i) % mb_chann->msg_buf_num;
mb_msg = mailbox_msg_ptr(mb_chann, slot);
/* Pairs with smp_store_release() in mailbox_msg_done(). */
if (!smp_load_acquire(&mb_msg->busy)) {
mb_chann->next_slot = (slot + 1) % mb_chann->msg_buf_num;
return slot | MAGIC_VAL;
}
}
return -ENOBUFS;
}
static void mailbox_release_msgid(struct mailbox_channel *mb_chann, int msg_id)
static struct mailbox_msg *mailbox_get_msg_buf(struct mailbox_channel *mb_chann,
size_t msg_size)
{
msg_id &= ~MAGIC_VAL_MASK;
xa_erase_irq(&mb_chann->chan_xa, msg_id);
}
struct mailbox_msg *mb_msg;
int msg_id;
u32 slot;
static void mailbox_release_msg(struct mailbox_channel *mb_chann,
struct mailbox_msg *mb_msg)
{
MB_DBG(mb_chann, "msg_id 0x%x msg opcode 0x%x",
mb_msg->pkg.header.id, mb_msg->pkg.header.opcode);
if (mb_msg->notify_cb)
mb_msg->notify_cb(mb_msg->handle, NULL, 0);
kfree(mb_msg);
/* msg_id = slot | MAGIC_VAL */
msg_id = mailbox_acquire_msgid(mb_chann);
if (msg_id < 0) {
MB_ERR(mb_chann, "No free message slot");
return NULL;
}
slot = msg_id & ~MAGIC_VAL_MASK;
mb_msg = mailbox_msg_ptr(mb_chann, slot);
memset(mb_msg, 0, msg_size + sizeof(*mb_msg));
mb_msg->busy = true;
mb_msg->pkg.header.id = msg_id;
return mb_msg;
}
static int
@@ -224,7 +273,7 @@ check_again:
if (tail < head && tmp_tail >= head) {
ret = read_poll_timeout(mailbox_get_headptr, head,
tmp_tail < head || tail >= head,
1, 100, false, mb_chann, CHAN_RES_X2I);
1, 2000000, false, mb_chann, CHAN_RES_X2I);
if (ret)
return ret;
@@ -248,8 +297,9 @@ mailbox_get_resp(struct mailbox_channel *mb_chann, struct xdna_msg_header *heade
void __iomem *data)
{
struct mailbox_msg *mb_msg;
int msg_id;
int ret = 0;
int msg_id;
u32 slot;
msg_id = header->id;
if (!mailbox_validate_msgid(msg_id)) {
@@ -257,22 +307,32 @@ mailbox_get_resp(struct mailbox_channel *mb_chann, struct xdna_msg_header *heade
return -EINVAL;
}
msg_id &= ~MAGIC_VAL_MASK;
mb_msg = xa_erase_irq(&mb_chann->chan_xa, msg_id);
if (!mb_msg) {
MB_ERR(mb_chann, "Cannot find msg 0x%x", msg_id);
slot = msg_id & ~MAGIC_VAL_MASK;
if (unlikely(slot >= mb_chann->msg_buf_num)) {
MB_ERR(mb_chann, "Invalid msg_id 0x%x", msg_id);
return -EINVAL;
}
mb_msg = mailbox_msg_ptr(mb_chann, slot);
MB_DBG(mb_chann, "opcode 0x%x size %d id 0x%x",
header->opcode, header->total_size, header->id);
if (mb_msg->notify_cb) {
ret = mb_msg->notify_cb(mb_msg->handle, data, header->total_size);
if (unlikely(ret))
MB_ERR(mb_chann, "Message callback ret %d", ret);
/*
* msg_id is slot | MAGIC_VAL and is reused as soon as the slot is
* freed. The management firmware protocol guarantees exactly one
* response for each request, so a response cannot refer to a previous
* user of a reused slot. Keep the busy check as a defensive guard
* against an unexpected response for an idle slot.
*/
if (unlikely(!mb_msg->busy)) {
MB_WARN_ONCE(mb_chann, "Unexpected response for idle slot 0x%x", msg_id);
return 0;
}
kfree(mb_msg);
ret = mailbox_msg_done(mb_msg, data, header->total_size);
if (unlikely(ret))
MB_ERR(mb_chann, "Message callback ret %d", ret);
return ret;
}
@@ -399,7 +459,7 @@ again:
}
int xdna_mailbox_send_msg(struct mailbox_channel *mb_chann,
const struct xdna_mailbox_msg *msg, u64 tx_timeout)
const struct xdna_mailbox_msg *msg)
{
struct xdna_msg_header *header;
struct mailbox_msg *mb_msg;
@@ -428,9 +488,12 @@ int xdna_mailbox_send_msg(struct mailbox_channel *mb_chann,
return -EPIPE;
}
mb_msg = kzalloc(sizeof(*mb_msg) + pkg_size, GFP_KERNEL);
if (!mb_msg)
return -ENOMEM;
mutex_lock(&mb_chann->lock);
mb_msg = mailbox_get_msg_buf(mb_chann, pkg_size);
if (!mb_msg) {
mutex_unlock(&mb_chann->lock);
return -ENOBUFS;
}
mb_msg->handle = msg->handle;
mb_msg->notify_cb = msg->notify_cb;
@@ -447,28 +510,18 @@ int xdna_mailbox_send_msg(struct mailbox_channel *mb_chann,
header->opcode = msg->opcode;
memcpy(mb_msg->pkg.payload, msg->send_data, msg->send_size);
ret = mailbox_acquire_msgid(mb_chann, mb_msg);
if (unlikely(ret < 0)) {
MB_ERR(mb_chann, "mailbox_acquire_msgid failed");
goto msg_id_failed;
}
header->id = ret;
MB_DBG(mb_chann, "opcode 0x%x size %d id 0x%x",
header->opcode, header->total_size, header->id);
ret = mailbox_send_msg(mb_chann, mb_msg);
if (ret) {
MB_DBG(mb_chann, "Error in mailbox send msg, ret %d", ret);
goto release_id;
mb_msg->busy = false;
mutex_unlock(&mb_chann->lock);
return ret;
}
mutex_unlock(&mb_chann->lock);
return 0;
release_id:
mailbox_release_msgid(mb_chann, header->id);
msg_id_failed:
kfree(mb_msg);
return ret;
}
@@ -481,12 +534,13 @@ struct mailbox_channel *xdna_mailbox_alloc_channel(struct mailbox *mb)
return NULL;
INIT_WORK(&mb_chann->rx_work, mailbox_rx_worker);
mb_chann->work_q = create_singlethread_workqueue(MAILBOX_NAME);
mb_chann->work_q = alloc_ordered_workqueue(MAILBOX_NAME, WQ_MEM_RECLAIM);
if (!mb_chann->work_q) {
MB_ERR(mb_chann, "Create workqueue failed");
goto free_chann;
}
mb_chann->mb = mb;
mutex_init(&mb_chann->lock);
return mb_chann;
@@ -500,7 +554,9 @@ void xdna_mailbox_free_channel(struct mailbox_channel *mb_chann)
if (!mb_chann)
return;
kfree(mb_chann->msg_buf);
destroy_workqueue(mb_chann->work_q);
mutex_destroy(&mb_chann->lock);
kfree(mb_chann);
}
@@ -509,21 +565,32 @@ xdna_mailbox_start_channel(struct mailbox_channel *mb_chann,
const struct xdna_mailbox_chann_res *x2i,
const struct xdna_mailbox_chann_res *i2x,
u32 iohub_int_addr,
int mb_irq)
int mb_irq, u32 n_msg)
{
int ret;
if (!is_power_of_2(x2i->rb_size) || !is_power_of_2(i2x->rb_size)) {
pr_err("Ring buf size must be power of 2\n");
return -EINVAL;
}
/*
* The ring buffer geometry, rb_start_addr and rb_size for both the
* x2i and the i2x channel, comes from AMD signed firmware, through
* the management or mailbox information block, or through the
* CREATE_CONTEXT response. The driver trusts those values by design
* and does not bound them against the mapped region.
*/
mb_chann->msix_irq = mb_irq;
mb_chann->iohub_int_addr = iohub_int_addr;
memcpy(&mb_chann->res[CHAN_RES_X2I], x2i, sizeof(*x2i));
memcpy(&mb_chann->res[CHAN_RES_I2X], i2x, sizeof(*i2x));
xa_init_flags(&mb_chann->chan_xa, XA_FLAGS_ALLOC | XA_FLAGS_LOCK_IRQ);
/*
* msg_buf is a flat array of n_msg slots, separate from the hardware
* ring buffer. Each slot is rb_size bytes - the combined size of the
* mailbox_msg metadata and the package payload must not exceed rb_size.
*/
mb_chann->msg_buf = kcalloc(n_msg, MSG_BUF_SZ(mb_chann), GFP_KERNEL);
if (!mb_chann->msg_buf)
return -ENOMEM;
mb_chann->msg_buf_num = n_msg;
mb_chann->x2i_tail = mailbox_get_tailptr(mb_chann, CHAN_RES_X2I);
mb_chann->i2x_head = mailbox_get_headptr(mb_chann, CHAN_RES_I2X);
@@ -531,6 +598,8 @@ xdna_mailbox_start_channel(struct mailbox_channel *mb_chann,
ret = request_irq(mb_irq, mailbox_irq_handler, 0, MAILBOX_NAME, mb_chann);
if (ret) {
MB_ERR(mb_chann, "Failed to request irq %d ret %d", mb_irq, ret);
kfree(mb_chann->msg_buf);
mb_chann->msg_buf = NULL;
return ret;
}
@@ -544,7 +613,7 @@ xdna_mailbox_start_channel(struct mailbox_channel *mb_chann,
void xdna_mailbox_stop_channel(struct mailbox_channel *mb_chann)
{
struct mailbox_msg *mb_msg;
unsigned long msg_id;
u32 i;
if (!mb_chann)
return;
@@ -555,12 +624,22 @@ void xdna_mailbox_stop_channel(struct mailbox_channel *mb_chann)
/* Cancel RX work and wait for it to finish */
drain_workqueue(mb_chann->work_q);
/* We can clean up and release resources */
xa_for_each_start(&mb_chann->chan_xa, msg_id, mb_msg, mb_chann->next_msgid)
mailbox_release_msg(mb_chann, mb_msg);
xa_for_each_range(&mb_chann->chan_xa, msg_id, mb_msg, 0, mb_chann->next_msgid - 1)
mailbox_release_msg(mb_chann, mb_msg);
xa_destroy(&mb_chann->chan_xa);
/* Release any in-flight messages in the pool
*
* Drain in submission order: next_slot points to the slot after
* the most recently assigned one, so start from next_slot and
* wrap around so callbacks fire oldest-first.
*/
for (i = 0; i < mb_chann->msg_buf_num; i++) {
u32 slot = (mb_chann->next_slot + i) % mb_chann->msg_buf_num;
mb_msg = mailbox_msg_ptr(mb_chann, slot);
if (!mb_msg->busy)
continue;
MB_DBG(mb_chann, "msg_id 0x%x msg opcode 0x%x",
mb_msg->pkg.header.id, mb_msg->pkg.header.opcode);
mailbox_msg_done(mb_msg, NULL, 0);
}
MB_DBG(mb_chann, "Mailbox channel stopped, irq: %d", mb_chann->msix_irq);
}
+6 -3
View File
@@ -88,6 +88,10 @@ struct mailbox_channel *xdna_mailbox_alloc_channel(struct mailbox *mb);
* @i2x: firmware to host mailbox resources
* @xdna_mailbox_intr_reg: register addr of MSI-X interrupt
* @mb_irq: Linux IRQ number associated with mailbox MSI-X interrupt vector index
* @n_msg: number of message slots to pre-allocate; must be non-zero for
* the PCI transport (the platform transport ignores this argument).
* Each slot is rb_size bytes. The caller must ensure at most n_msg
* messages are in flight at any time.
*
* Return: If success, return a handle of mailbox channel. Otherwise, return NULL.
*/
@@ -96,7 +100,7 @@ xdna_mailbox_start_channel(struct mailbox_channel *mb_chann,
const struct xdna_mailbox_chann_res *x2i,
const struct xdna_mailbox_chann_res *i2x,
u32 xdna_mailbox_intr_reg,
int mb_irq);
int mb_irq, u32 n_msg);
/*
* xdna_mailbox_free_channel() -- free mailbox channel
@@ -117,11 +121,10 @@ void xdna_mailbox_stop_channel(struct mailbox_channel *mailbox_chann);
*
* @mailbox_chann: Mailbox channel handle
* @msg: message struct for message information
* @tx_timeout: the timeout value for sending the message in ms.
*
* Return: If success return 0, otherwise, return error code
*/
int xdna_mailbox_send_msg(struct mailbox_channel *mailbox_chann,
const struct xdna_mailbox_msg *msg, u64 tx_timeout);
const struct xdna_mailbox_msg *msg);
#endif /* _AIE_MAILBOX_ */
@@ -44,7 +44,7 @@ int xdna_send_msg_wait(struct amdxdna_dev *xdna, struct mailbox_channel *chann,
struct xdna_notify *hdl = msg->handle;
int ret;
ret = xdna_mailbox_send_msg(chann, msg, TX_TIMEOUT);
ret = xdna_mailbox_send_msg(chann, msg);
if (ret) {
XDNA_ERR(xdna, "Send message failed, ret %d", ret);
return ret;
@@ -6,7 +6,6 @@
#ifndef _AMDXDNA_MAILBOX_HELPER_H
#define _AMDXDNA_MAILBOX_HELPER_H
#define TX_TIMEOUT 2000 /* milliseconds */
#define RX_TIMEOUT 5000 /* milliseconds */
struct amdxdna_dev;
+6
View File
@@ -14,6 +14,7 @@
#include <linux/iommu.h>
#include <linux/pci.h>
#include "aie.h"
#include "amdxdna_cbuf.h"
#include "amdxdna_ctx.h"
#include "amdxdna_debugfs.h"
@@ -151,6 +152,8 @@ static int amdxdna_drm_open(struct drm_device *ddev, struct drm_file *filp)
filp->driver_priv = client;
client->filp = filp;
spin_lock_init(&client->io_stats.lock);
XDNA_DBG(xdna, "pid %d opened", client->pid);
return 0;
@@ -287,6 +290,9 @@ static void amdxdna_show_fdinfo(struct drm_printer *p, struct drm_file *filp)
size_t heap_usage, external_usage, internal_usage;
char *drv_name = filp->minor->dev->driver->name;
drm_printf(p, "drm-engine-npu:\t%llu ns\n",
amdxdna_io_stats_busy_time_ns(client));
mutex_lock(&client->mm_lock);
heap_usage = client->heap_usage;
+9
View File
@@ -143,6 +143,13 @@ struct amdxdna_device_id {
const struct amdxdna_dev_info *dev_info;
};
struct amdxdna_io_stats {
spinlock_t lock; /* protect io stats */
int job_depth;
u64 start_time;
u64 busy_time;
};
/*
* struct amdxdna_client - amdxdna client
* A per fd data structure for managing context and other user process stuffs.
@@ -169,6 +176,8 @@ struct amdxdna_client {
size_t heap_usage;
size_t total_bo_usage;
size_t total_int_bo_usage;
struct amdxdna_io_stats io_stats;
};
#define amdxdna_for_each_hwctx(client, hwctx_id, entry) \
+1
View File
@@ -153,6 +153,7 @@ const struct amdxdna_rev_vbnv npu4_rev_vbnv_tbl[] = {
{ AIE2_DEV_REVISION_GPT1, "NPU Gorgon Point 1" },
{ AIE2_DEV_REVISION_GPT2, "NPU Gorgon Point 2" },
{ AIE2_DEV_REVISION_GPT3, "NPU Gorgon Point 3" },
{ AIE2_DEV_REVISION_GRGH, "NPU Gorgon Halo" },
{ 0 }
};
+1 -1
View File
@@ -12,10 +12,10 @@
#include <drm/drm_drv.h>
#include <drm/drm_ioctl.h>
#include <drm/drm_utils.h>
#include <drm/drm_gem.h>
#include <drm/drm_accel.h>
#include <drm/drm_managed.h>
#include <drm/drm_timeout.h>
#include <drm/ethosu_accel.h>
#include "ethosu_drv.h"
+1 -3
View File
@@ -345,10 +345,8 @@ int ethosu_job_init(struct ethosu_device *edev)
ethosu_job_irq_handler_thread,
0, KBUILD_MODNAME,
edev);
if (ret) {
dev_err(dev, "failed to request irq\n");
if (ret)
return ret;
}
edev->fence_context = dma_fence_context_alloc(1);
+77 -21
View File
@@ -67,6 +67,48 @@ bool ivpu_force_snoop;
module_param_named(force_snoop, ivpu_force_snoop, bool, 0444);
MODULE_PARM_DESC(force_snoop, "Force snooping for NPU host memory access");
static struct pci_device_id ivpu_pci_ids[] = {
{ PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_MTL) },
{ PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_ARL) },
{ PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_LNL) },
{ PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_PTL) },
{ PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_WCL) },
{ PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_NVL) },
{}
};
MODULE_DEVICE_TABLE(pci, ivpu_pci_ids);
static int ivpu_ip_gen_init(struct ivpu_device *vdev)
{
vdev->platform = IVPU_PLATFORM_INVALID;
switch (ivpu_device_id(vdev)) {
case PCI_DEVICE_ID_MTL:
case PCI_DEVICE_ID_ARL:
vdev->hw_ip_gen = IVPU_HW_IP_37XX;
vdev->btrs_gen = IVPU_HW_BTRS_MTL;
break;
case PCI_DEVICE_ID_LNL:
vdev->hw_ip_gen = IVPU_HW_IP_40XX;
vdev->btrs_gen = IVPU_HW_BTRS_LNL;
break;
case PCI_DEVICE_ID_PTL:
case PCI_DEVICE_ID_WCL:
vdev->hw_ip_gen = IVPU_HW_IP_50XX;
vdev->btrs_gen = IVPU_HW_BTRS_LNL;
break;
case PCI_DEVICE_ID_NVL:
vdev->hw_ip_gen = IVPU_HW_IP_60XX;
vdev->btrs_gen = IVPU_HW_BTRS_LNL;
break;
default:
ivpu_err(vdev, "Unknown PCI device [%04x:%04x]\n", ivpu_vendor_id(vdev),
ivpu_device_id(vdev));
return -EINVAL;
}
return 0;
}
static struct ivpu_user_limits *ivpu_user_limits_alloc(struct ivpu_device *vdev, uid_t uid)
{
struct ivpu_user_limits *limits;
@@ -205,6 +247,7 @@ bool ivpu_is_capable(struct ivpu_device *vdev, u32 capability)
case DRM_IVPU_CAP_BO_CREATE_FROM_USERPTR:
return true;
case DRM_IVPU_CAP_MANAGE_CMDQ:
case DRM_IVPU_CAP_CMDQ_SET_PRIORITY:
return vdev->fw->sched_mode == VPU_SCHEDULING_MODE_HW;
default:
return false;
@@ -247,10 +290,7 @@ static int ivpu_get_param_ioctl(struct drm_device *dev, void *data, struct drm_f
break;
case DRM_IVPU_PARAM_FW_API_VERSION:
if (args->index < VPU_FW_API_VER_NUM) {
struct vpu_firmware_header *fw_hdr;
fw_hdr = (struct vpu_firmware_header *)vdev->fw->file->data;
args->value = fw_hdr->api_version[args->index];
args->value = vdev->fw->api_version[args->index];
} else {
ret = -EINVAL;
}
@@ -273,6 +313,9 @@ static int ivpu_get_param_ioctl(struct drm_device *dev, void *data, struct drm_f
case DRM_IVPU_PARAM_PREEMPT_BUFFER_SIZE:
args->value = ivpu_fw_preempt_buf_size(vdev);
break;
case DRM_IVPU_PARAM_CMDQ_PRIORITY:
ret = ivpu_cmdq_get_priority(file_priv, args->index, &args->value);
break;
default:
ret = -EINVAL;
break;
@@ -284,14 +327,23 @@ static int ivpu_get_param_ioctl(struct drm_device *dev, void *data, struct drm_f
static int ivpu_set_param_ioctl(struct drm_device *dev, void *data, struct drm_file *file)
{
struct ivpu_file_priv *file_priv = file->driver_priv;
struct drm_ivpu_param *args = data;
int ret = 0;
int idx;
if (!drm_dev_enter(dev, &idx))
return -ENODEV;
switch (args->param) {
case DRM_IVPU_PARAM_CMDQ_PRIORITY:
ret = ivpu_cmdq_set_priority(file_priv, args->index, args->value);
break;
default:
ret = -EINVAL;
}
drm_dev_exit(idx);
return ret;
}
@@ -468,6 +520,14 @@ int ivpu_boot(struct ivpu_device *vdev)
ivpu_fw_boot_params_setup(vdev, ivpu_bo_vaddr(vdev->fw->mem_bp));
vdev->fw->last_boot_mode = vdev->fw->next_boot_mode;
if (!ivpu_fw_is_warm_boot(vdev)) {
ret = ivpu_fw_load(vdev);
if (ret) {
ivpu_err(vdev, "Failed to load the firmware: %d\n", ret);
return ret;
}
}
ret = ivpu_hw_boot_fw(vdev);
if (ret) {
ivpu_err(vdev, "Failed to start the firmware: %d\n", ret);
@@ -484,6 +544,9 @@ int ivpu_boot(struct ivpu_device *vdev)
ivpu_hw_irq_enable(vdev);
ivpu_ipc_enable(vdev);
if (ivpu_test_mode & IVPU_TEST_MODE_FW_TEST)
return 0;
if (!ivpu_fw_is_warm_boot(vdev)) {
ret = ivpu_pm_dct_init(vdev);
if (ret)
@@ -637,6 +700,7 @@ static int ivpu_pci_init(struct ivpu_device *vdev)
struct pci_dev *pdev = to_pci_dev(vdev->drm.dev);
struct resource *bar0 = &pdev->resource[0];
struct resource *bar4 = &pdev->resource[4];
int dma_bits;
int ret;
ivpu_dbg(vdev, MISC, "Mapping BAR0 (RegV) %pR\n", bar0);
@@ -653,7 +717,12 @@ static int ivpu_pci_init(struct ivpu_device *vdev)
return PTR_ERR(vdev->regb);
}
ret = dma_set_mask_and_coherent(vdev->drm.dev, DMA_BIT_MASK(vdev->hw->dma_bits));
if (ivpu_hw_ip_gen(vdev) >= IVPU_HW_IP_40XX)
dma_bits = 42;
else
dma_bits = 38;
ret = dma_set_mask_and_coherent(vdev->drm.dev, DMA_BIT_MASK(dma_bits));
if (ret) {
ivpu_err(vdev, "Failed to set DMA mask: %d\n", ret);
return ret;
@@ -701,12 +770,10 @@ static int ivpu_dev_init(struct ivpu_device *vdev)
if (!vdev->pm)
return -ENOMEM;
if (ivpu_hw_ip_gen(vdev) >= IVPU_HW_IP_40XX)
vdev->hw->dma_bits = 48;
else
vdev->hw->dma_bits = 38;
ret = ivpu_ip_gen_init(vdev);
if (ret)
return ret;
vdev->platform = IVPU_PLATFORM_INVALID;
vdev->context_xa_limit.min = IVPU_USER_CONTEXT_MIN_SSID;
vdev->context_xa_limit.max = IVPU_USER_CONTEXT_MAX_SSID;
atomic64_set(&vdev->unique_id_counter, 0);
@@ -839,17 +906,6 @@ static void ivpu_dev_fini(struct ivpu_device *vdev)
xa_destroy(&vdev->context_xa);
}
static struct pci_device_id ivpu_pci_ids[] = {
{ PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_MTL) },
{ PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_ARL) },
{ PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_LNL) },
{ PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_PTL_P) },
{ PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_WCL) },
{ PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_NVL) },
{}
};
MODULE_DEVICE_TABLE(pci, ivpu_pci_ids);
static int ivpu_probe(struct pci_dev *pdev, const struct pci_device_id *id)
{
struct ivpu_device *vdev;
+29 -43
View File
@@ -27,21 +27,13 @@
#define PCI_DEVICE_ID_MTL 0x7d1d
#define PCI_DEVICE_ID_ARL 0xad1d
#define PCI_DEVICE_ID_LNL 0x643e
#define PCI_DEVICE_ID_PTL_P 0xb03e
#define PCI_DEVICE_ID_PTL 0xb03e
#define PCI_DEVICE_ID_WCL 0xfd3e
#define PCI_DEVICE_ID_NVL 0xd71d
#define IVPU_HW_IP_37XX 37
#define IVPU_HW_IP_40XX 40
#define IVPU_HW_IP_50XX 50
#define IVPU_HW_IP_60XX 60
#define IVPU_HW_IP_REV_LNL_B0 4
#define IVPU_HW_IP_REV_NVL_A0 0
#define IVPU_HW_BTRS_MTL 1
#define IVPU_HW_BTRS_LNL 2
#define IVPU_GLOBAL_CONTEXT_MMU_SSID 0
/* SSID 1 is used by the VPU to represent reserved context */
#define IVPU_RESERVED_CONTEXT_MMU_SSID 1
@@ -91,7 +83,7 @@
drm_warn(&(vdev)->drm, "%s(): " fmt, __func__, ##__VA_ARGS__)
#define ivpu_warn_ratelimited(vdev, fmt, ...) \
drm_err_ratelimited(&(vdev)->drm, "%s(): " fmt, __func__, ##__VA_ARGS__)
drm_warn_ratelimited(&(vdev)->drm, "%s(): " fmt, __func__, ##__VA_ARGS__)
#define ivpu_info(vdev, fmt, ...) drm_info(&(vdev)->drm, fmt, ##__VA_ARGS__)
@@ -107,10 +99,21 @@
ivpu_dbg(vdev, MISC, "Using WA: " #wa_name "\n"); \
} while (0)
enum vpu_hw_ip {
IVPU_HW_IP_37XX = 37,
IVPU_HW_IP_40XX = 40,
IVPU_HW_IP_50XX = 50,
IVPU_HW_IP_60XX = 60,
};
enum vpu_hw_btrs {
IVPU_HW_BTRS_MTL = 1,
IVPU_HW_BTRS_LNL = 2,
};
struct ivpu_wa_table {
bool punit_disabled;
bool clear_runtime_mem;
bool interrupt_clear_with_0;
bool disable_clock_relinquish;
bool wp0_during_power_up;
bool disable_d0i2;
@@ -136,6 +139,8 @@ struct ivpu_device {
struct drm_device drm;
void __iomem *regb;
void __iomem *regv;
enum vpu_hw_ip hw_ip_gen;
enum vpu_hw_btrs btrs_gen;
u32 platform;
u32 irq;
@@ -186,6 +191,9 @@ struct ivpu_device {
int autosuspend;
int d0i3_entry_msg;
int state_dump_msg;
int mmu_reg;
int mmu_queue;
int pwr_island_status;
} timeout;
};
@@ -222,6 +230,7 @@ extern bool ivpu_force_snoop;
#define IVPU_TEST_MODE_FW_TEST BIT(0)
#define IVPU_TEST_MODE_NULL_HW BIT(1)
#define IVPU_TEST_MODE_NULL_SUBMISSION BIT(2)
#define IVPU_TEST_MODE_FREQ_CHANGE_ENABLE BIT(3)
#define IVPU_TEST_MODE_MIP_DISABLE BIT(6)
#define IVPU_TEST_MODE_DISABLE_TIMEOUTS BIT(8)
#define IVPU_TEST_MODE_TURBO_ENABLE BIT(9)
@@ -244,47 +253,24 @@ static inline u8 ivpu_revision(struct ivpu_device *vdev)
return to_pci_dev(vdev->drm.dev)->revision;
}
static inline u16 ivpu_vendor_id(struct ivpu_device *vdev)
{
return to_pci_dev(vdev->drm.dev)->vendor;
}
static inline u16 ivpu_device_id(struct ivpu_device *vdev)
{
return to_pci_dev(vdev->drm.dev)->device;
}
static inline int ivpu_hw_ip_gen(struct ivpu_device *vdev)
static inline enum vpu_hw_ip ivpu_hw_ip_gen(struct ivpu_device *vdev)
{
switch (ivpu_device_id(vdev)) {
case PCI_DEVICE_ID_MTL:
case PCI_DEVICE_ID_ARL:
return IVPU_HW_IP_37XX;
case PCI_DEVICE_ID_LNL:
return IVPU_HW_IP_40XX;
case PCI_DEVICE_ID_PTL_P:
case PCI_DEVICE_ID_WCL:
return IVPU_HW_IP_50XX;
case PCI_DEVICE_ID_NVL:
return IVPU_HW_IP_60XX;
default:
dump_stack();
ivpu_err(vdev, "Unknown NPU IP generation\n");
return 0;
}
return vdev->hw_ip_gen;
}
static inline int ivpu_hw_btrs_gen(struct ivpu_device *vdev)
static inline enum vpu_hw_btrs ivpu_hw_btrs_gen(struct ivpu_device *vdev)
{
switch (ivpu_device_id(vdev)) {
case PCI_DEVICE_ID_MTL:
case PCI_DEVICE_ID_ARL:
return IVPU_HW_BTRS_MTL;
case PCI_DEVICE_ID_LNL:
case PCI_DEVICE_ID_PTL_P:
case PCI_DEVICE_ID_WCL:
case PCI_DEVICE_ID_NVL:
return IVPU_HW_BTRS_LNL;
default:
dump_stack();
ivpu_err(vdev, "Unknown buttress generation\n");
return 0;
}
return vdev->btrs_gen;
}
static inline struct ivpu_device *to_ivpu_device(struct drm_device *dev)
+47 -59
View File
@@ -28,15 +28,15 @@
#define ADDR_TO_L2_CACHE_CFG(addr) ((addr) >> 31)
/* Check if FW API is compatible with the driver */
#define IVPU_FW_CHECK_API_COMPAT(vdev, fw_hdr, name, min_major) \
ivpu_fw_check_api(vdev, fw_hdr, #name, \
#define IVPU_FW_CHECK_API_COMPAT(vdev, name, min_major) \
ivpu_fw_check_api(vdev, #name, \
VPU_##name##_API_VER_INDEX, \
VPU_##name##_API_VER_MAJOR, \
VPU_##name##_API_VER_MINOR, min_major)
/* Check if API version is lower that the given version */
#define IVPU_FW_CHECK_API_VER_LT(vdev, fw_hdr, name, major, minor) \
ivpu_fw_check_api_ver_lt(vdev, fw_hdr, #name, VPU_##name##_API_VER_INDEX, major, minor)
#define IVPU_FW_CHECK_API_VER_LT(vdev, name, major, minor) \
ivpu_fw_check_api_ver_lt(vdev, #name, VPU_##name##_API_VER_INDEX, major, minor)
#define IVPU_FOCUS_PRESENT_TIMER_MS 1000
@@ -93,12 +93,11 @@ static int ivpu_fw_request(struct ivpu_device *vdev)
}
static int
ivpu_fw_check_api(struct ivpu_device *vdev, const struct vpu_firmware_header *fw_hdr,
const char *str, int index, u16 expected_major, u16 expected_minor,
u16 min_major)
ivpu_fw_check_api(struct ivpu_device *vdev, const char *str, int index, u16 expected_major,
u16 expected_minor, u16 min_major)
{
u16 major = (u16)(fw_hdr->api_version[index] >> 16);
u16 minor = (u16)(fw_hdr->api_version[index]);
u16 major = (u16)(vdev->fw->api_version[index] >> 16);
u16 minor = (u16)(vdev->fw->api_version[index]);
if (major < min_major) {
ivpu_err(vdev, "Incompatible FW %s API version: %d.%d, required %d.0 or later\n",
@@ -116,11 +115,10 @@ ivpu_fw_check_api(struct ivpu_device *vdev, const struct vpu_firmware_header *fw
}
static bool
ivpu_fw_check_api_ver_lt(struct ivpu_device *vdev, const struct vpu_firmware_header *fw_hdr,
const char *str, int index, u16 major, u16 minor)
ivpu_fw_check_api_ver_lt(struct ivpu_device *vdev, const char *str, int index, u16 major, u16 minor)
{
u16 fw_major = (u16)(fw_hdr->api_version[index] >> 16);
u16 fw_minor = (u16)(fw_hdr->api_version[index]);
u16 fw_major = (u16)(vdev->fw->api_version[index] >> 16);
u16 fw_minor = (u16)(vdev->fw->api_version[index]);
if (fw_major < major || (fw_major == major && fw_minor < minor))
return true;
@@ -142,7 +140,7 @@ bool ivpu_is_within_range(u64 addr, size_t size, struct ivpu_addr_range *range)
}
static u32
ivpu_fw_sched_mode_select(struct ivpu_device *vdev, const struct vpu_firmware_header *fw_hdr)
ivpu_fw_sched_mode_select(struct ivpu_device *vdev)
{
if (ivpu_hw_ip_gen(vdev) >= IVPU_HW_IP_60XX &&
ivpu_sched_mode == VPU_SCHEDULING_MODE_OS) {
@@ -153,7 +151,7 @@ ivpu_fw_sched_mode_select(struct ivpu_device *vdev, const struct vpu_firmware_he
if (ivpu_sched_mode != IVPU_SCHED_MODE_AUTO)
return ivpu_sched_mode;
if (IVPU_FW_CHECK_API_VER_LT(vdev, fw_hdr, JSM, 3, 24))
if (IVPU_FW_CHECK_API_VER_LT(vdev, JSM, 3, 24))
return VPU_SCHEDULING_MODE_OS;
return VPU_SCHEDULING_MODE_HW;
@@ -302,9 +300,11 @@ static int ivpu_fw_parse(struct ivpu_device *vdev)
ivpu_info(vdev, "Firmware: %s, version: %s\n", fw->name, fw->version);
if (IVPU_FW_CHECK_API_COMPAT(vdev, fw_hdr, BOOT, 3))
memcpy(fw->api_version, fw_hdr->api_version, sizeof(fw->api_version));
if (IVPU_FW_CHECK_API_COMPAT(vdev, BOOT, 3))
return -EINVAL;
if (IVPU_FW_CHECK_API_COMPAT(vdev, fw_hdr, JSM, 3))
if (IVPU_FW_CHECK_API_COMPAT(vdev, JSM, 3))
return -EINVAL;
fw->boot_params_addr = boot_params_addr;
@@ -324,7 +324,7 @@ static int ivpu_fw_parse(struct ivpu_device *vdev)
fw->dvfs_mode = 0;
fw->sched_mode = ivpu_fw_sched_mode_select(vdev, fw_hdr);
fw->sched_mode = ivpu_fw_sched_mode_select(vdev);
ivpu_info(vdev, "Scheduler mode: %s\n", fw->sched_mode ? "HW" : "OS");
ivpu_preemption_config_parse(vdev, fw_hdr);
@@ -364,6 +364,25 @@ static void ivpu_fw_release(struct ivpu_device *vdev)
}
static void ivpu_fw_mem_fini(struct ivpu_device *vdev)
{
struct ivpu_fw_info *fw = vdev->fw;
ivpu_bo_free(fw->mem_shave_nn);
ivpu_bo_free(fw->mem_log_verb);
ivpu_bo_free(fw->mem_log_crit);
ivpu_bo_free(fw->mem);
ivpu_bo_free(fw->mem_fw_ver);
ivpu_bo_free(fw->mem_bp);
fw->mem_shave_nn = NULL;
fw->mem_log_verb = NULL;
fw->mem_log_crit = NULL;
fw->mem = NULL;
fw->mem_fw_ver = NULL;
fw->mem_bp = NULL;
}
static int ivpu_fw_mem_init(struct ivpu_device *vdev)
{
struct ivpu_fw_info *fw = vdev->fw;
@@ -382,7 +401,7 @@ static int ivpu_fw_mem_init(struct ivpu_device *vdev)
if (!fw->mem_fw_ver) {
ivpu_err(vdev, "Failed to create firmware version memory buffer\n");
ret = -ENOMEM;
goto err_free_bp;
goto err_free;
}
fw->mem = ivpu_bo_create_runtime(vdev, fw->runtime_addr, fw->runtime_size,
@@ -390,14 +409,14 @@ static int ivpu_fw_mem_init(struct ivpu_device *vdev)
if (!fw->mem) {
ivpu_err(vdev, "Failed to create firmware runtime memory buffer\n");
ret = -ENOMEM;
goto err_free_fw_ver;
goto err_free;
}
ret = ivpu_mmu_context_set_pages_ro(vdev, &vdev->gctx, fw->read_only_addr,
fw->read_only_size);
if (ret) {
ivpu_err(vdev, "Failed to set firmware image read-only\n");
goto err_free_fw_mem;
goto err_free;
}
fw->mem_log_crit = ivpu_bo_create_global(vdev, IVPU_FW_CRITICAL_BUFFER_SIZE,
@@ -405,7 +424,7 @@ static int ivpu_fw_mem_init(struct ivpu_device *vdev)
if (!fw->mem_log_crit) {
ivpu_err(vdev, "Failed to create critical log buffer\n");
ret = -ENOMEM;
goto err_free_fw_mem;
goto err_free;
}
if (ivpu_fw_log_level <= IVPU_FW_LOG_INFO)
@@ -418,7 +437,7 @@ static int ivpu_fw_mem_init(struct ivpu_device *vdev)
if (!fw->mem_log_verb) {
ivpu_err(vdev, "Failed to create verbose log buffer\n");
ret = -ENOMEM;
goto err_free_log_crit;
goto err_free;
}
if (fw->shave_nn_size) {
@@ -427,47 +446,17 @@ static int ivpu_fw_mem_init(struct ivpu_device *vdev)
if (!fw->mem_shave_nn) {
ivpu_err(vdev, "Failed to create shavenn buffer\n");
ret = -ENOMEM;
goto err_free_log_verb;
goto err_free;
}
}
return 0;
err_free_log_verb:
ivpu_bo_free(fw->mem_log_verb);
err_free_log_crit:
ivpu_bo_free(fw->mem_log_crit);
err_free_fw_mem:
ivpu_bo_free(fw->mem);
err_free_fw_ver:
ivpu_bo_free(fw->mem_fw_ver);
err_free_bp:
ivpu_bo_free(fw->mem_bp);
err_free:
ivpu_fw_mem_fini(vdev);
return ret;
}
static void ivpu_fw_mem_fini(struct ivpu_device *vdev)
{
struct ivpu_fw_info *fw = vdev->fw;
if (fw->mem_shave_nn) {
ivpu_bo_free(fw->mem_shave_nn);
fw->mem_shave_nn = NULL;
}
ivpu_bo_free(fw->mem_log_verb);
ivpu_bo_free(fw->mem_log_crit);
ivpu_bo_free(fw->mem);
ivpu_bo_free(fw->mem_fw_ver);
ivpu_bo_free(fw->mem_bp);
fw->mem_log_verb = NULL;
fw->mem_log_crit = NULL;
fw->mem = NULL;
fw->mem_fw_ver = NULL;
fw->mem_bp = NULL;
}
int ivpu_fw_init(struct ivpu_device *vdev)
{
int ret;
@@ -484,8 +473,6 @@ int ivpu_fw_init(struct ivpu_device *vdev)
if (ret)
goto err_fw_release;
ivpu_fw_load(vdev);
return 0;
err_fw_release:
@@ -499,7 +486,7 @@ void ivpu_fw_fini(struct ivpu_device *vdev)
ivpu_fw_release(vdev);
}
void ivpu_fw_load(struct ivpu_device *vdev)
int ivpu_fw_load(struct ivpu_device *vdev)
{
struct ivpu_fw_info *fw = vdev->fw;
u64 image_end_offset = fw->image_load_offset + fw->image_size;
@@ -516,6 +503,7 @@ void ivpu_fw_load(struct ivpu_device *vdev)
}
wmb(); /* Flush WC buffers after writing fw->mem */
return 0;
}
static void ivpu_fw_boot_params_print(struct ivpu_device *vdev, struct vpu_boot_params *boot_params)
+2 -1
View File
@@ -20,6 +20,7 @@ struct ivpu_fw_info {
const struct firmware *file;
const char *name;
char version[FW_VERSION_STR_SIZE];
u32 api_version[VPU_FW_API_VER_NUM];
struct ivpu_bo *mem_bp;
struct ivpu_bo *mem_fw_ver;
struct ivpu_bo *mem;
@@ -54,7 +55,7 @@ struct ivpu_fw_info {
bool ivpu_is_within_range(u64 addr, size_t size, struct ivpu_addr_range *range);
int ivpu_fw_init(struct ivpu_device *vdev);
void ivpu_fw_fini(struct ivpu_device *vdev);
void ivpu_fw_load(struct ivpu_device *vdev);
int ivpu_fw_load(struct ivpu_device *vdev);
void ivpu_fw_boot_params_setup(struct ivpu_device *vdev, struct vpu_boot_params *boot_params);
static inline bool ivpu_fw_is_warm_boot(struct ivpu_device *vdev)
+4 -1
View File
@@ -12,7 +12,7 @@
#include <drm/drm_cache.h>
#include <drm/drm_debugfs.h>
#include <drm/drm_file.h>
#include <drm/drm_utils.h>
#include <drm/drm_timeout.h>
#include "ivpu_drv.h"
#include "ivpu_fw.h"
@@ -471,6 +471,9 @@ struct ivpu_bo *ivpu_bo_create_global(struct ivpu_device *vdev, u64 size, u32 fl
void ivpu_bo_free(struct ivpu_bo *bo)
{
if (!bo)
return;
struct iosys_map map = IOSYS_MAP_INIT_VADDR(bo->base.vaddr);
if (bo->flags & DRM_IVPU_BO_MAPPABLE) {
+23 -10
View File
@@ -67,9 +67,6 @@ static void wa_init(struct ivpu_device *vdev)
vdev->wa.punit_disabled = false;
vdev->wa.clear_runtime_mem = false;
if (ivpu_hw_btrs_gen(vdev) == IVPU_HW_BTRS_MTL)
vdev->wa.interrupt_clear_with_0 = ivpu_hw_btrs_irqs_clear_with_0_mtl(vdev);
if ((ivpu_device_id(vdev) == PCI_DEVICE_ID_LNL &&
ivpu_revision(vdev) < IVPU_HW_IP_REV_LNL_B0) ||
(ivpu_device_id(vdev) == PCI_DEVICE_ID_NVL &&
@@ -90,7 +87,6 @@ static void wa_init(struct ivpu_device *vdev)
IVPU_PRINT_WA(punit_disabled);
IVPU_PRINT_WA(clear_runtime_mem);
IVPU_PRINT_WA(interrupt_clear_with_0);
IVPU_PRINT_WA(disable_clock_relinquish);
IVPU_PRINT_WA(wp0_during_power_up);
IVPU_PRINT_WA(disable_d0i2);
@@ -98,14 +94,22 @@ static void wa_init(struct ivpu_device *vdev)
static void timeouts_init(struct ivpu_device *vdev)
{
if (ivpu_test_mode & IVPU_TEST_MODE_DISABLE_TIMEOUTS) {
if (ivpu_test_mode & (IVPU_TEST_MODE_DISABLE_TIMEOUTS | IVPU_TEST_MODE_FW_TEST)) {
/* in milliseconds, negative value disables the timeout */
vdev->timeout.boot = -1;
vdev->timeout.jsm = -1;
vdev->timeout.tdr = -1;
vdev->timeout.inference = -1;
vdev->timeout.autosuspend = -1;
vdev->timeout.d0i3_entry_msg = -1;
/* fixed delay, not a timeout - has to remain finite */
vdev->timeout.state_dump_msg = 100;
/* in microseconds, zero disables the timeout */
vdev->timeout.mmu_reg = 0;
vdev->timeout.mmu_queue = 0;
vdev->timeout.pwr_island_status = 0;
} else if (ivpu_is_fpga(vdev)) {
/* in milliseconds */
vdev->timeout.boot = 50;
vdev->timeout.jsm = 15000;
vdev->timeout.tdr = 30000;
@@ -113,7 +117,12 @@ static void timeouts_init(struct ivpu_device *vdev)
vdev->timeout.autosuspend = -1;
vdev->timeout.d0i3_entry_msg = 500;
vdev->timeout.state_dump_msg = 10000;
/* in microseconds */
vdev->timeout.mmu_reg = 100 * USEC_PER_MSEC;
vdev->timeout.mmu_queue = 1000 * USEC_PER_MSEC;
vdev->timeout.pwr_island_status = 5 * USEC_PER_MSEC;
} else if (ivpu_is_simics(vdev)) {
/* in milliseconds */
vdev->timeout.boot = 50;
vdev->timeout.jsm = 500;
vdev->timeout.tdr = 10000;
@@ -121,7 +130,12 @@ static void timeouts_init(struct ivpu_device *vdev)
vdev->timeout.autosuspend = 100;
vdev->timeout.d0i3_entry_msg = 100;
vdev->timeout.state_dump_msg = 10;
/* in microseconds */
vdev->timeout.mmu_reg = 10 * USEC_PER_MSEC;
vdev->timeout.mmu_queue = 100 * USEC_PER_MSEC;
vdev->timeout.pwr_island_status = 5 * USEC_PER_MSEC;
} else {
/* in milliseconds */
vdev->timeout.boot = 1000;
vdev->timeout.jsm = 500;
vdev->timeout.tdr = 2000;
@@ -132,6 +146,10 @@ static void timeouts_init(struct ivpu_device *vdev)
vdev->timeout.autosuspend = 100;
vdev->timeout.d0i3_entry_msg = 5;
vdev->timeout.state_dump_msg = 100;
/* in microseconds */
vdev->timeout.mmu_reg = 10 * USEC_PER_MSEC;
vdev->timeout.mmu_queue = 100 * USEC_PER_MSEC;
vdev->timeout.pwr_island_status = 5 * USEC_PER_MSEC;
}
}
@@ -356,11 +374,6 @@ void ivpu_hw_profiling_freq_drive(struct ivpu_device *vdev, bool enable)
void ivpu_irq_handlers_init(struct ivpu_device *vdev)
{
if (ivpu_hw_ip_gen(vdev) == IVPU_HW_IP_37XX)
vdev->hw->irq.ip_irq_handler = ivpu_hw_ip_irq_handler_37xx;
else
vdev->hw->irq.ip_irq_handler = ivpu_hw_ip_irq_handler_40xx;
if (ivpu_hw_btrs_gen(vdev) == IVPU_HW_BTRS_MTL)
vdev->hw->irq.btrs_irq_handler = ivpu_hw_btrs_irq_handler_mtl;
else
+1 -7
View File
@@ -18,7 +18,6 @@ struct ivpu_addr_range {
struct ivpu_hw_info {
struct {
bool (*btrs_irq_handler)(struct ivpu_device *vdev, int irq);
bool (*ip_irq_handler)(struct ivpu_device *vdev, int irq);
} irq;
struct {
struct ivpu_addr_range runtime;
@@ -49,9 +48,9 @@ struct ivpu_hw_info {
u32 tile_fuse;
u32 sku;
u16 config;
int dma_bits;
ktime_t d0i3_entry_host_ts;
u64 d0i3_entry_vpu_ts;
u32 btrs_irq_mask;
atomic_t firewall_irq_counter;
};
@@ -74,11 +73,6 @@ static inline u32 ivpu_hw_btrs_irq_handler(struct ivpu_device *vdev, int irq)
return vdev->hw->irq.btrs_irq_handler(vdev, irq);
}
static inline u32 ivpu_hw_ip_irq_handler(struct ivpu_device *vdev, int irq)
{
return vdev->hw->irq.ip_irq_handler(vdev, irq);
}
static inline u64 ivpu_hw_range_size(const struct ivpu_addr_range *range)
{
return range->end - range->start;
-195
View File
@@ -1,195 +0,0 @@
/* SPDX-License-Identifier: GPL-2.0-only */
/*
* Copyright (C) 2020-2023 Intel Corporation
*/
#ifndef __IVPU_HW_40XX_REG_H__
#define __IVPU_HW_40XX_REG_H__
#include <linux/bits.h>
#define VPU_40XX_HOST_SS_CPR_CLK_EN 0x00000080u
#define VPU_40XX_HOST_SS_CPR_CLK_EN_TOP_NOC_MASK BIT_MASK(1)
#define VPU_40XX_HOST_SS_CPR_CLK_EN_DSS_MAS_MASK BIT_MASK(10)
#define VPU_40XX_HOST_SS_CPR_CLK_EN_CSS_MAS_MASK BIT_MASK(11)
#define VPU_40XX_HOST_SS_CPR_CLK_SET 0x00000084u
#define VPU_40XX_HOST_SS_CPR_CLK_SET_TOP_NOC_MASK BIT_MASK(1)
#define VPU_40XX_HOST_SS_CPR_CLK_SET_DSS_MAS_MASK BIT_MASK(10)
#define VPU_40XX_HOST_SS_CPR_CLK_SET_MSS_MAS_MASK BIT_MASK(11)
#define VPU_40XX_HOST_SS_CPR_RST_EN 0x00000090u
#define VPU_40XX_HOST_SS_CPR_RST_EN_TOP_NOC_MASK BIT_MASK(1)
#define VPU_40XX_HOST_SS_CPR_RST_EN_DSS_MAS_MASK BIT_MASK(10)
#define VPU_40XX_HOST_SS_CPR_RST_EN_CSS_MAS_MASK BIT_MASK(11)
#define VPU_40XX_HOST_SS_CPR_RST_SET 0x00000094u
#define VPU_40XX_HOST_SS_CPR_RST_SET_TOP_NOC_MASK BIT_MASK(1)
#define VPU_40XX_HOST_SS_CPR_RST_SET_DSS_MAS_MASK BIT_MASK(10)
#define VPU_40XX_HOST_SS_CPR_RST_SET_MSS_MAS_MASK BIT_MASK(11)
#define VPU_40XX_HOST_SS_CPR_RST_CLR 0x00000098u
#define VPU_40XX_HOST_SS_CPR_RST_CLR_TOP_NOC_MASK BIT_MASK(1)
#define VPU_40XX_HOST_SS_CPR_RST_CLR_DSS_MAS_MASK BIT_MASK(10)
#define VPU_40XX_HOST_SS_CPR_RST_CLR_MSS_MAS_MASK BIT_MASK(11)
#define VPU_40XX_HOST_SS_HW_VERSION 0x00000108u
#define VPU_40XX_HOST_SS_HW_VERSION_SOC_REVISION_MASK GENMASK(7, 0)
#define VPU_40XX_HOST_SS_HW_VERSION_SOC_NUMBER_MASK GENMASK(15, 8)
#define VPU_40XX_HOST_SS_HW_VERSION_VPU_GENERATION_MASK GENMASK(23, 16)
#define VPU_40XX_HOST_SS_SW_VERSION 0x0000010cu
#define VPU_40XX_HOST_SS_GEN_CTRL 0x00000118u
#define VPU_40XX_HOST_SS_GEN_CTRL_PS_MASK GENMASK(31, 29)
#define VPU_40XX_HOST_SS_NOC_QREQN 0x00000154u
#define VPU_40XX_HOST_SS_NOC_QREQN_TOP_SOCMMIO_MASK BIT_MASK(0)
#define VPU_40XX_HOST_SS_NOC_QACCEPTN 0x00000158u
#define VPU_40XX_HOST_SS_NOC_QACCEPTN_TOP_SOCMMIO_MASK BIT_MASK(0)
#define VPU_40XX_HOST_SS_NOC_QDENY 0x0000015cu
#define VPU_40XX_HOST_SS_NOC_QDENY_TOP_SOCMMIO_MASK BIT_MASK(0)
#define VPU_40XX_TOP_NOC_QREQN 0x00000160u
#define VPU_40XX_TOP_NOC_QREQN_CPU_CTRL_MASK BIT_MASK(0)
#define VPU_40XX_TOP_NOC_QREQN_HOSTIF_L2CACHE_MASK BIT_MASK(2)
#define VPU_40XX_TOP_NOC_QACCEPTN 0x00000164u
#define VPU_40XX_TOP_NOC_QACCEPTN_CPU_CTRL_MASK BIT_MASK(0)
#define VPU_40XX_TOP_NOC_QACCEPTN_HOSTIF_L2CACHE_MASK BIT_MASK(2)
#define VPU_40XX_TOP_NOC_QDENY 0x00000168u
#define VPU_40XX_TOP_NOC_QDENY_CPU_CTRL_MASK BIT_MASK(0)
#define VPU_40XX_TOP_NOC_QDENY_HOSTIF_L2CACHE_MASK BIT_MASK(2)
#define VPU_40XX_HOST_SS_FW_SOC_IRQ_EN 0x00000170u
#define VPU_40XX_HOST_SS_FW_SOC_IRQ_EN_CSS_ROM_CMX_MASK BIT_MASK(0)
#define VPU_40XX_HOST_SS_FW_SOC_IRQ_EN_CSS_DBG_MASK BIT_MASK(1)
#define VPU_40XX_HOST_SS_FW_SOC_IRQ_EN_CSS_CTRL_MASK BIT_MASK(2)
#define VPU_40XX_HOST_SS_FW_SOC_IRQ_EN_DEC400_MASK BIT_MASK(3)
#define VPU_40XX_HOST_SS_FW_SOC_IRQ_EN_MSS_NCE_MASK BIT_MASK(4)
#define VPU_40XX_HOST_SS_FW_SOC_IRQ_EN_MSS_MBI_MASK BIT_MASK(5)
#define VPU_40XX_HOST_SS_FW_SOC_IRQ_EN_MSS_MBI_CMX_MASK BIT_MASK(6)
#define VPU_40XX_HOST_SS_ICB_STATUS_0 0x00010210u
#define VPU_40XX_HOST_SS_ICB_STATUS_0_TIMER_0_INT_MASK BIT_MASK(0)
#define VPU_40XX_HOST_SS_ICB_STATUS_0_TIMER_1_INT_MASK BIT_MASK(1)
#define VPU_40XX_HOST_SS_ICB_STATUS_0_TIMER_2_INT_MASK BIT_MASK(2)
#define VPU_40XX_HOST_SS_ICB_STATUS_0_TIMER_3_INT_MASK BIT_MASK(3)
#define VPU_40XX_HOST_SS_ICB_STATUS_0_HOST_IPC_FIFO_INT_MASK BIT_MASK(4)
#define VPU_40XX_HOST_SS_ICB_STATUS_0_MMU_IRQ_0_INT_MASK BIT_MASK(5)
#define VPU_40XX_HOST_SS_ICB_STATUS_0_MMU_IRQ_1_INT_MASK BIT_MASK(6)
#define VPU_40XX_HOST_SS_ICB_STATUS_0_MMU_IRQ_2_INT_MASK BIT_MASK(7)
#define VPU_40XX_HOST_SS_ICB_STATUS_0_NOC_FIREWALL_INT_MASK BIT_MASK(8)
#define VPU_40XX_HOST_SS_ICB_STATUS_0_CPU_INT_REDIRECT_0_INT_MASK BIT_MASK(30)
#define VPU_40XX_HOST_SS_ICB_STATUS_0_CPU_INT_REDIRECT_1_INT_MASK BIT_MASK(31)
#define VPU_40XX_HOST_SS_ICB_STATUS_1 0x00010214u
#define VPU_40XX_HOST_SS_ICB_STATUS_1_CPU_INT_REDIRECT_2_INT_MASK BIT_MASK(0)
#define VPU_40XX_HOST_SS_ICB_STATUS_1_CPU_INT_REDIRECT_3_INT_MASK BIT_MASK(1)
#define VPU_40XX_HOST_SS_ICB_STATUS_1_CPU_INT_REDIRECT_4_INT_MASK BIT_MASK(2)
#define VPU_40XX_HOST_SS_ICB_CLEAR_0 0x00010220u
#define VPU_40XX_HOST_SS_ICB_CLEAR_1 0x00010224u
#define VPU_40XX_HOST_SS_ICB_ENABLE_0 0x00010240u
#define VPU_40XX_HOST_SS_ICB_ENABLE_1 0x00010244u
#define VPU_40XX_HOST_SS_TIM_IPC_FIFO_ATM 0x000200f4u
#define VPU_40XX_HOST_SS_TIM_IPC_FIFO_STAT 0x000200fcu
#define VPU_40XX_HOST_SS_TIM_IPC_FIFO_STAT_FILL_LEVEL_MASK GENMASK(23, 16)
#define VPU_40XX_HOST_SS_AON_PWR_ISO_EN0 0x00030020u
#define VPU_40XX_HOST_SS_AON_PWR_ISO_EN0_CSS_CPU_MASK BIT_MASK(3)
#define VPU_40XX_HOST_SS_AON_PWR_ISLAND_EN0 0x00030024u
#define VPU_40XX_HOST_SS_AON_PWR_ISLAND_EN0_CSS_CPU_MASK BIT_MASK(3)
#define VPU_40XX_HOST_SS_AON_PWR_ISLAND_TRICKLE_EN0 0x00030028u
#define VPU_40XX_HOST_SS_AON_PWR_ISLAND_TRICKLE_EN0_CSS_CPU_MASK BIT_MASK(3)
#define VPU_40XX_HOST_SS_AON_PWR_ISLAND_STATUS0 0x0003002cu
#define VPU_40XX_HOST_SS_AON_PWR_ISLAND_STATUS0_CSS_CPU_MASK BIT_MASK(3)
#define VPU_50XX_HOST_SS_AON_PWR_ISLAND_EN_POST_DLY 0x00030068u
#define VPU_50XX_HOST_SS_AON_PWR_ISLAND_EN_POST_DLY_POST_DLY_MASK GENMASK(7, 0)
#define VPU_50XX_HOST_SS_AON_PWR_ISLAND_EN_POST_DLY_POST1_DLY_MASK GENMASK(15, 8)
#define VPU_50XX_HOST_SS_AON_PWR_ISLAND_EN_POST_DLY_POST2_DLY_MASK GENMASK(23, 16)
#define VPU_50XX_HOST_SS_AON_PWR_ISLAND_STATUS_DLY 0x0003006cu
#define VPU_50XX_HOST_SS_AON_PWR_ISLAND_STATUS_DLY_STATUS_DLY_MASK GENMASK(7, 0)
#define VPU_40XX_HOST_SS_AON_IDLE_GEN 0x00030200u
#define VPU_40XX_HOST_SS_AON_IDLE_GEN_EN_MASK BIT_MASK(0)
#define VPU_40XX_HOST_SS_AON_IDLE_GEN_HW_PG_EN_MASK BIT_MASK(1)
#define VPU_40XX_HOST_SS_AON_DPU_ACTIVE 0x00030204u
#define VPU_40XX_HOST_SS_AON_DPU_ACTIVE_DPU_ACTIVE_MASK BIT_MASK(0)
#define VPU_50XX_HOST_SS_AON_FABRIC_REQ_OVERRIDE 0x00030210u
#define VPU_50XX_HOST_SS_AON_FABRIC_REQ_OVERRIDE_REQ_OVERRIDE_MASK BIT_MASK(0)
#define VPU_40XX_HOST_SS_VERIFICATION_ADDRESS_LO 0x00040040u
#define VPU_40XX_HOST_SS_VERIFICATION_ADDRESS_LO_DONE_MASK BIT_MASK(0)
#define VPU_40XX_HOST_SS_VERIFICATION_ADDRESS_LO_IOSF_RS_ID_MASK GENMASK(2, 1)
#define VPU_40XX_HOST_SS_VERIFICATION_ADDRESS_LO_IMAGE_LOCATION_MASK GENMASK(31, 3)
#define VPU_40XX_HOST_SS_WORKPOINT_CONFIG_MIRROR 0x00082020u
#define VPU_40XX_HOST_SS_WORKPOINT_CONFIG_MIRROR_FINAL_PLL_FREQ_MASK GENMASK(15, 0)
#define VPU_40XX_HOST_SS_WORKPOINT_CONFIG_MIRROR_CONFIG_ID_MASK GENMASK(31, 16)
#define VPU_40XX_HOST_IF_TCU_PTW_OVERRIDES 0x00360000u
#define VPU_40XX_HOST_IF_TCU_PTW_OVERRIDES_CACHE_OVERRIDE_EN_MASK BIT_MASK(0)
#define VPU_40XX_HOST_IF_TCU_PTW_OVERRIDES_AWCACHE_OVERRIDE_MASK BIT_MASK(1)
#define VPU_40XX_HOST_IF_TCU_PTW_OVERRIDES_ARCACHE_OVERRIDE_MASK BIT_MASK(2)
#define VPU_40XX_HOST_IF_TCU_PTW_OVERRIDES_SNOOP_OVERRIDE_EN_MASK BIT_MASK(3)
#define VPU_40XX_HOST_IF_TCU_PTW_OVERRIDES_AW_SNOOP_OVERRIDE_MASK BIT_MASK(4)
#define VPU_40XX_HOST_IF_TCU_PTW_OVERRIDES_AR_SNOOP_OVERRIDE_MASK BIT_MASK(5)
#define VPU_40XX_HOST_IF_TCU_PTW_OVERRIDES_PTW_AW_CONTEXT_FLAG_MASK GENMASK(10, 6)
#define VPU_40XX_HOST_IF_TCU_PTW_OVERRIDES_PTW_AR_CONTEXT_FLAG_MASK GENMASK(15, 11)
#define VPU_40XX_HOST_IF_TBU_MMUSSIDV 0x00360004u
#define VPU_40XX_HOST_IF_TBU_MMUSSIDV_TBU0_AWMMUSSIDV_MASK BIT_MASK(0)
#define VPU_40XX_HOST_IF_TBU_MMUSSIDV_TBU0_ARMMUSSIDV_MASK BIT_MASK(1)
#define VPU_40XX_HOST_IF_TBU_MMUSSIDV_TBU1_AWMMUSSIDV_MASK BIT_MASK(2)
#define VPU_40XX_HOST_IF_TBU_MMUSSIDV_TBU1_ARMMUSSIDV_MASK BIT_MASK(3)
#define VPU_40XX_HOST_IF_TBU_MMUSSIDV_TBU2_AWMMUSSIDV_MASK BIT_MASK(4)
#define VPU_40XX_HOST_IF_TBU_MMUSSIDV_TBU2_ARMMUSSIDV_MASK BIT_MASK(5)
#define VPU_40XX_HOST_IF_TBU_MMUSSIDV_TBU3_AWMMUSSIDV_MASK BIT_MASK(6)
#define VPU_40XX_HOST_IF_TBU_MMUSSIDV_TBU3_ARMMUSSIDV_MASK BIT_MASK(7)
#define VPU_40XX_HOST_IF_TBU_MMUSSIDV_TBU4_AWMMUSSIDV_MASK BIT_MASK(8)
#define VPU_40XX_HOST_IF_TBU_MMUSSIDV_TBU4_ARMMUSSIDV_MASK BIT_MASK(9)
#define VPU_40XX_CPU_SS_DSU_LEON_RT_BASE 0x04000000u
#define VPU_40XX_CPU_SS_DSU_LEON_RT_DSU_CTRL 0x04000000u
#define VPU_40XX_CPU_SS_DSU_LEON_RT_PC_REG 0x04400010u
#define VPU_40XX_CPU_SS_DSU_LEON_RT_NPC_REG 0x04400014u
#define VPU_40XX_CPU_SS_DSU_LEON_RT_DSU_TRAP_REG 0x04400020u
#define VPU_40XX_CPU_SS_TIM_WATCHDOG 0x0102009cu
#define VPU_40XX_CPU_SS_TIM_WDOG_EN 0x010200a4u
#define VPU_40XX_CPU_SS_TIM_SAFE 0x010200a8u
#define VPU_40XX_CPU_SS_TIM_GEN_CONFIG 0x01021008u
#define VPU_40XX_CPU_SS_TIM_GEN_CONFIG_WDOG_TO_INT_CLR_MASK BIT_MASK(9)
#define VPU_40XX_CPU_SS_CPR_NOC_QREQN 0x01010030u
#define VPU_40XX_CPU_SS_CPR_NOC_QREQN_TOP_MMIO_MASK BIT_MASK(0)
#define VPU_40XX_CPU_SS_CPR_NOC_QACCEPTN 0x01010034u
#define VPU_40XX_CPU_SS_CPR_NOC_QACCEPTN_TOP_MMIO_MASK BIT_MASK(0)
#define VPU_40XX_CPU_SS_CPR_NOC_QDENY 0x01010038u
#define VPU_40XX_CPU_SS_CPR_NOC_QDENY_TOP_MMIO_MASK BIT_MASK(0)
#define VPU_40XX_CPU_SS_TIM_IPC_FIFO 0x010200f0u
#define VPU_40XX_CPU_SS_TIM_PERF_EXT_FREE_CNT 0x01029008u
#define VPU_40XX_CPU_SS_DOORBELL_0 0x01300000u
#define VPU_40XX_CPU_SS_DOORBELL_0_SET_MASK BIT_MASK(0)
#define VPU_40XX_CPU_SS_DOORBELL_1 0x01301000u
#endif /* __IVPU_HW_40XX_REG_H__ */
+18 -38
View File
@@ -8,8 +8,7 @@
#include "ivpu_drv.h"
#include "ivpu_hw.h"
#include "ivpu_hw_btrs.h"
#include "ivpu_hw_btrs_lnl_reg.h"
#include "ivpu_hw_btrs_mtl_reg.h"
#include "ivpu_hw_btrs_reg.h"
#include "ivpu_hw_reg_io.h"
#include "ivpu_jsm_msg.h"
#include "ivpu_pm.h"
@@ -24,9 +23,6 @@
(REG_FLD(VPU_HW_BTRS_LNL_INTERRUPT_STAT, IMR1_ERR)) | \
(REG_FLD(VPU_HW_BTRS_LNL_INTERRUPT_STAT, SURV_ERR)))
#define BTRS_MTL_ALL_IRQ_MASK (BTRS_MTL_IRQ_MASK | (REG_FLD(VPU_HW_BTRS_MTL_INTERRUPT_STAT, \
FREQ_CHANGE)))
#define BTRS_IRQ_DISABLE_MASK ((u32)-1)
#define BTRS_LNL_ALL_IRQ_MASK ((u32)-1)
@@ -49,9 +45,6 @@
#define BTRS_MTL_TILE_FUSE_ENABLE_BOTH 0x0
#define BTRS_MTL_TILE_SKU_BOTH 0x3630
#define BTRS_LNL_TILE_MAX_NUM 6
#define BTRS_LNL_TILE_MAX_MASK 0x3f
#define WEIGHTS_DEFAULT 0xf711f711u
#define WEIGHTS_ATS_DEFAULT 0x0000f711u
@@ -59,18 +52,6 @@
#define DCT_ENABLE 0x1
#define DCT_DISABLE 0x0
int ivpu_hw_btrs_irqs_clear_with_0_mtl(struct ivpu_device *vdev)
{
REGB_WR32(VPU_HW_BTRS_MTL_INTERRUPT_STAT, BTRS_MTL_ALL_IRQ_MASK);
if (REGB_RD32(VPU_HW_BTRS_MTL_INTERRUPT_STAT) == BTRS_MTL_ALL_IRQ_MASK) {
/* Writing 1s does not clear the interrupt status register */
REGB_WR32(VPU_HW_BTRS_MTL_INTERRUPT_STAT, 0x0);
return true;
}
return false;
}
static void freq_ratios_init_mtl(struct ivpu_device *vdev)
{
struct ivpu_hw_info *hw = vdev->hw;
@@ -115,13 +96,9 @@ void ivpu_hw_btrs_freq_ratios_init(struct ivpu_device *vdev)
static bool tile_disable_check(u32 config)
{
/* Allowed values: 0 or one bit from range 0-5 (6 tiles) */
if (config == 0)
return true;
if (config > BIT(BTRS_LNL_TILE_MAX_NUM - 1))
return false;
if ((config & (config - 1)) == 0)
return true;
@@ -156,6 +133,11 @@ static int info_init_mtl(struct ivpu_device *vdev)
hw->tile_fuse = BTRS_MTL_TILE_FUSE_ENABLE_BOTH;
hw->sku = BTRS_MTL_TILE_SKU_BOTH;
hw->config = WP_CONFIG(MTL_CONFIG_2_TILE, MTL_PLL_RATIO_4_3);
hw->btrs_irq_mask = BTRS_MTL_IRQ_MASK;
if (ivpu_test_mode & IVPU_TEST_MODE_FREQ_CHANGE_ENABLE)
hw->btrs_irq_mask = REG_SET_FLD(VPU_HW_BTRS_MTL_INTERRUPT_STAT, FREQ_CHANGE,
hw->btrs_irq_mask);
return 0;
}
@@ -172,6 +154,11 @@ static int info_init_lnl(struct ivpu_device *vdev)
hw->tile_fuse = tile_fuse_config;
hw->pll.profiling_freq = PLL_PROFILING_FREQ_DEFAULT;
hw->btrs_irq_mask = BTRS_LNL_IRQ_MASK;
if (ivpu_test_mode & IVPU_TEST_MODE_FREQ_CHANGE_ENABLE)
hw->btrs_irq_mask = REG_SET_FLD(VPU_HW_BTRS_LNL_INTERRUPT_STAT, FREQ_CHANGE,
hw->btrs_irq_mask);
return 0;
}
@@ -698,7 +685,7 @@ int ivpu_hw_btrs_cfg_freq_init(struct ivpu_device *vdev)
/* Handler for IRQs from Buttress core (irqB) */
bool ivpu_hw_btrs_irq_handler_mtl(struct ivpu_device *vdev, int irq)
{
u32 status = REGB_RD32(VPU_HW_BTRS_MTL_INTERRUPT_STAT) & BTRS_MTL_IRQ_MASK;
u32 status = REGB_RD32(VPU_HW_BTRS_MTL_INTERRUPT_STAT) & vdev->hw->btrs_irq_mask;
bool schedule_recovery = false;
if (!status)
@@ -729,14 +716,7 @@ bool ivpu_hw_btrs_irq_handler_mtl(struct ivpu_device *vdev, int irq)
}
/* This must be done after interrupts are cleared at the source. */
if (IVPU_WA(interrupt_clear_with_0))
/*
* Writing 1 triggers an interrupt, so we can't perform read update write.
* Clear local interrupt status by writing 0 to all bits.
*/
REGB_WR32(VPU_HW_BTRS_MTL_INTERRUPT_STAT, 0x0);
else
REGB_WR32(VPU_HW_BTRS_MTL_INTERRUPT_STAT, status);
REGB_WR32(VPU_HW_BTRS_MTL_INTERRUPT_STAT, status);
if (schedule_recovery)
ivpu_pm_trigger_recovery(vdev, "Buttress IRQ");
@@ -747,7 +727,7 @@ bool ivpu_hw_btrs_irq_handler_mtl(struct ivpu_device *vdev, int irq)
/* Handler for IRQs from Buttress core (irqB) */
bool ivpu_hw_btrs_irq_handler_lnl(struct ivpu_device *vdev, int irq)
{
u32 status = REGB_RD32(VPU_HW_BTRS_LNL_INTERRUPT_STAT) & BTRS_LNL_IRQ_MASK;
u32 status = REGB_RD32(VPU_HW_BTRS_LNL_INTERRUPT_STAT) & vdev->hw->btrs_irq_mask;
bool schedule_recovery = false;
if (!status)
@@ -889,10 +869,10 @@ void ivpu_hw_btrs_global_int_enable(struct ivpu_device *vdev)
void ivpu_hw_btrs_irq_enable(struct ivpu_device *vdev)
{
if (ivpu_hw_btrs_gen(vdev) == IVPU_HW_BTRS_MTL) {
REGB_WR32(VPU_HW_BTRS_MTL_LOCAL_INT_MASK, (u32)(~BTRS_MTL_IRQ_MASK));
REGB_WR32(VPU_HW_BTRS_MTL_LOCAL_INT_MASK, (u32)(~vdev->hw->btrs_irq_mask));
REGB_WR32(VPU_HW_BTRS_MTL_GLOBAL_INT_MASK, 0x0);
} else {
REGB_WR32(VPU_HW_BTRS_LNL_LOCAL_INT_MASK, (u32)(~BTRS_LNL_IRQ_MASK));
REGB_WR32(VPU_HW_BTRS_LNL_LOCAL_INT_MASK, (u32)(~vdev->hw->btrs_irq_mask));
REGB_WR32(VPU_HW_BTRS_LNL_GLOBAL_INT_MASK, 0x0);
}
}
@@ -910,7 +890,7 @@ void ivpu_hw_btrs_irq_disable(struct ivpu_device *vdev)
static void diagnose_failure_mtl(struct ivpu_device *vdev)
{
u32 reg = REGB_RD32(VPU_HW_BTRS_MTL_INTERRUPT_STAT) & BTRS_MTL_IRQ_MASK;
u32 reg = REGB_RD32(VPU_HW_BTRS_MTL_INTERRUPT_STAT) & vdev->hw->btrs_irq_mask;
if (REG_TEST_FLD(VPU_HW_BTRS_MTL_INTERRUPT_STAT, ATS_ERR, reg))
ivpu_err(vdev, "ATS_ERR irq 0x%016llx", REGB_RD64(VPU_HW_BTRS_MTL_ATS_ERR_LOG_0));
@@ -927,7 +907,7 @@ static void diagnose_failure_mtl(struct ivpu_device *vdev)
static void diagnose_failure_lnl(struct ivpu_device *vdev)
{
u32 reg = REGB_RD32(VPU_HW_BTRS_LNL_INTERRUPT_STAT) & BTRS_LNL_IRQ_MASK;
u32 reg = REGB_RD32(VPU_HW_BTRS_LNL_INTERRUPT_STAT) & vdev->hw->btrs_irq_mask;
if (REG_TEST_FLD(VPU_HW_BTRS_LNL_INTERRUPT_STAT, ATS_ERR, reg)) {
ivpu_err(vdev, "ATS_ERR_LOG1 0x%08x ATS_ERR_LOG2 0x%08x\n",
-3
View File
@@ -7,8 +7,6 @@
#define __IVPU_HW_BTRS_H__
#include "ivpu_drv.h"
#include "ivpu_hw_37xx_reg.h"
#include "ivpu_hw_40xx_reg.h"
#include "ivpu_hw_reg_io.h"
#define PLL_PROFILING_FREQ_DEFAULT 38400000
@@ -19,7 +17,6 @@
int ivpu_hw_btrs_info_init(struct ivpu_device *vdev);
void ivpu_hw_btrs_freq_ratios_init(struct ivpu_device *vdev);
int ivpu_hw_btrs_irqs_clear_with_0_mtl(struct ivpu_device *vdev);
int ivpu_hw_btrs_wp_drive(struct ivpu_device *vdev, bool enable);
int ivpu_hw_btrs_wait_for_clock_res_own_ack(struct ivpu_device *vdev);
int ivpu_hw_btrs_d0i3_enable(struct ivpu_device *vdev);
-83
View File
@@ -1,83 +0,0 @@
/* SPDX-License-Identifier: GPL-2.0-only */
/*
* Copyright (C) 2020-2023 Intel Corporation
*/
#ifndef __IVPU_HW_BTRS_MTL_REG_H__
#define __IVPU_HW_BTRS_MTL_REG_H__
#include <linux/bits.h>
#define VPU_HW_BTRS_MTL_INTERRUPT_TYPE 0x00000000u
#define VPU_HW_BTRS_MTL_INTERRUPT_STAT 0x00000004u
#define VPU_HW_BTRS_MTL_INTERRUPT_STAT_FREQ_CHANGE_MASK BIT_MASK(0)
#define VPU_HW_BTRS_MTL_INTERRUPT_STAT_ATS_ERR_MASK BIT_MASK(1)
#define VPU_HW_BTRS_MTL_INTERRUPT_STAT_UFI_ERR_MASK BIT_MASK(2)
#define VPU_HW_BTRS_MTL_WP_REQ_PAYLOAD0 0x00000008u
#define VPU_HW_BTRS_MTL_WP_REQ_PAYLOAD0_MIN_RATIO_MASK GENMASK(15, 0)
#define VPU_HW_BTRS_MTL_WP_REQ_PAYLOAD0_MAX_RATIO_MASK GENMASK(31, 16)
#define VPU_HW_BTRS_MTL_WP_REQ_PAYLOAD1 0x0000000cu
#define VPU_HW_BTRS_MTL_WP_REQ_PAYLOAD1_TARGET_RATIO_MASK GENMASK(15, 0)
#define VPU_HW_BTRS_MTL_WP_REQ_PAYLOAD1_EPP_MASK GENMASK(31, 16)
#define VPU_HW_BTRS_MTL_WP_REQ_PAYLOAD2 0x00000010u
#define VPU_HW_BTRS_MTL_WP_REQ_PAYLOAD2_CONFIG_MASK GENMASK(15, 0)
#define VPU_HW_BTRS_MTL_WP_REQ_CMD 0x00000014u
#define VPU_HW_BTRS_MTL_WP_REQ_CMD_SEND_MASK BIT_MASK(0)
#define VPU_HW_BTRS_MTL_WP_DOWNLOAD 0x00000018u
#define VPU_HW_BTRS_MTL_WP_DOWNLOAD_TARGET_RATIO_MASK GENMASK(15, 0)
#define VPU_HW_BTRS_MTL_CURRENT_PLL 0x0000001cu
#define VPU_HW_BTRS_MTL_CURRENT_PLL_RATIO_MASK GENMASK(15, 0)
#define VPU_HW_BTRS_MTL_PLL_ENABLE 0x00000020u
#define VPU_HW_BTRS_MTL_FMIN_FUSE 0x00000024u
#define VPU_HW_BTRS_MTL_FMIN_FUSE_MIN_RATIO_MASK GENMASK(7, 0)
#define VPU_HW_BTRS_MTL_FMIN_FUSE_PN_RATIO_MASK GENMASK(15, 8)
#define VPU_HW_BTRS_MTL_FMAX_FUSE 0x00000028u
#define VPU_HW_BTRS_MTL_FMAX_FUSE_MAX_RATIO_MASK GENMASK(7, 0)
#define VPU_HW_BTRS_MTL_TILE_FUSE 0x0000002cu
#define VPU_HW_BTRS_MTL_TILE_FUSE_VALID_MASK BIT_MASK(0)
#define VPU_HW_BTRS_MTL_TILE_FUSE_SKU_MASK GENMASK(3, 2)
#define VPU_HW_BTRS_MTL_LOCAL_INT_MASK 0x00000030u
#define VPU_HW_BTRS_MTL_GLOBAL_INT_MASK 0x00000034u
#define VPU_HW_BTRS_MTL_PLL_STATUS 0x00000040u
#define VPU_HW_BTRS_MTL_PLL_STATUS_LOCK_MASK BIT_MASK(1)
#define VPU_HW_BTRS_MTL_VPU_STATUS 0x00000044u
#define VPU_HW_BTRS_MTL_VPU_STATUS_READY_MASK BIT_MASK(0)
#define VPU_HW_BTRS_MTL_VPU_STATUS_IDLE_MASK BIT_MASK(1)
#define VPU_HW_BTRS_MTL_VPU_D0I3_CONTROL 0x00000060u
#define VPU_HW_BTRS_MTL_VPU_D0I3_CONTROL_INPROGRESS_MASK BIT_MASK(0)
#define VPU_HW_BTRS_MTL_VPU_D0I3_CONTROL_I3_MASK BIT_MASK(2)
#define VPU_HW_BTRS_MTL_VPU_IP_RESET 0x00000050u
#define VPU_HW_BTRS_MTL_VPU_IP_RESET_TRIGGER_MASK BIT_MASK(0)
#define VPU_HW_BTRS_MTL_VPU_TELEMETRY_OFFSET 0x00000080u
#define VPU_HW_BTRS_MTL_VPU_TELEMETRY_SIZE 0x00000084u
#define VPU_HW_BTRS_MTL_VPU_TELEMETRY_ENABLE 0x00000088u
#define VPU_HW_BTRS_MTL_ATS_ERR_LOG_0 0x000000a0u
#define VPU_HW_BTRS_MTL_ATS_ERR_LOG_1 0x000000a4u
#define VPU_HW_BTRS_MTL_ATS_ERR_CLEAR 0x000000a8u
#define VPU_HW_BTRS_MTL_UFI_ERR_LOG 0x000000b0u
#define VPU_HW_BTRS_MTL_UFI_ERR_LOG_CQ_ID_MASK GENMASK(11, 0)
#define VPU_HW_BTRS_MTL_UFI_ERR_LOG_AXI_ID_MASK GENMASK(19, 12)
#define VPU_HW_BTRS_MTL_UFI_ERR_LOG_OPCODE_MASK GENMASK(24, 20)
#define VPU_HW_BTRS_MTL_UFI_ERR_CLEAR 0x000000b4u
#endif /* __IVPU_HW_BTRS_MTL_REG_H__ */
@@ -1,13 +1,85 @@
/* SPDX-License-Identifier: GPL-2.0-only */
/*
* Copyright (C) 2020-2024 Intel Corporation
* Copyright (C) 2020-2026 Intel Corporation
*/
#ifndef __IVPU_HW_BTRS_LNL_REG_H__
#define __IVPU_HW_BTRS_LNL_REG_H__
#ifndef __IVPU_HW_BTRS_REG_H__
#define __IVPU_HW_BTRS_REG_H__
#include <linux/bits.h>
#define VPU_HW_BTRS_MTL_INTERRUPT_TYPE 0x00000000u
#define VPU_HW_BTRS_MTL_INTERRUPT_STAT 0x00000004u
#define VPU_HW_BTRS_MTL_INTERRUPT_STAT_FREQ_CHANGE_MASK BIT_MASK(0)
#define VPU_HW_BTRS_MTL_INTERRUPT_STAT_ATS_ERR_MASK BIT_MASK(1)
#define VPU_HW_BTRS_MTL_INTERRUPT_STAT_UFI_ERR_MASK BIT_MASK(2)
#define VPU_HW_BTRS_MTL_WP_REQ_PAYLOAD0 0x00000008u
#define VPU_HW_BTRS_MTL_WP_REQ_PAYLOAD0_MIN_RATIO_MASK GENMASK(15, 0)
#define VPU_HW_BTRS_MTL_WP_REQ_PAYLOAD0_MAX_RATIO_MASK GENMASK(31, 16)
#define VPU_HW_BTRS_MTL_WP_REQ_PAYLOAD1 0x0000000cu
#define VPU_HW_BTRS_MTL_WP_REQ_PAYLOAD1_TARGET_RATIO_MASK GENMASK(15, 0)
#define VPU_HW_BTRS_MTL_WP_REQ_PAYLOAD1_EPP_MASK GENMASK(31, 16)
#define VPU_HW_BTRS_MTL_WP_REQ_PAYLOAD2 0x00000010u
#define VPU_HW_BTRS_MTL_WP_REQ_PAYLOAD2_CONFIG_MASK GENMASK(15, 0)
#define VPU_HW_BTRS_MTL_WP_REQ_CMD 0x00000014u
#define VPU_HW_BTRS_MTL_WP_REQ_CMD_SEND_MASK BIT_MASK(0)
#define VPU_HW_BTRS_MTL_WP_DOWNLOAD 0x00000018u
#define VPU_HW_BTRS_MTL_WP_DOWNLOAD_TARGET_RATIO_MASK GENMASK(15, 0)
#define VPU_HW_BTRS_MTL_CURRENT_PLL 0x0000001cu
#define VPU_HW_BTRS_MTL_CURRENT_PLL_RATIO_MASK GENMASK(15, 0)
#define VPU_HW_BTRS_MTL_PLL_ENABLE 0x00000020u
#define VPU_HW_BTRS_MTL_FMIN_FUSE 0x00000024u
#define VPU_HW_BTRS_MTL_FMIN_FUSE_MIN_RATIO_MASK GENMASK(7, 0)
#define VPU_HW_BTRS_MTL_FMIN_FUSE_PN_RATIO_MASK GENMASK(15, 8)
#define VPU_HW_BTRS_MTL_FMAX_FUSE 0x00000028u
#define VPU_HW_BTRS_MTL_FMAX_FUSE_MAX_RATIO_MASK GENMASK(7, 0)
#define VPU_HW_BTRS_MTL_TILE_FUSE 0x0000002cu
#define VPU_HW_BTRS_MTL_TILE_FUSE_VALID_MASK BIT_MASK(0)
#define VPU_HW_BTRS_MTL_TILE_FUSE_SKU_MASK GENMASK(3, 2)
#define VPU_HW_BTRS_MTL_LOCAL_INT_MASK 0x00000030u
#define VPU_HW_BTRS_MTL_GLOBAL_INT_MASK 0x00000034u
#define VPU_HW_BTRS_MTL_PLL_STATUS 0x00000040u
#define VPU_HW_BTRS_MTL_PLL_STATUS_LOCK_MASK BIT_MASK(1)
#define VPU_HW_BTRS_MTL_VPU_STATUS 0x00000044u
#define VPU_HW_BTRS_MTL_VPU_STATUS_READY_MASK BIT_MASK(0)
#define VPU_HW_BTRS_MTL_VPU_STATUS_IDLE_MASK BIT_MASK(1)
#define VPU_HW_BTRS_MTL_VPU_IP_RESET 0x00000050u
#define VPU_HW_BTRS_MTL_VPU_IP_RESET_TRIGGER_MASK BIT_MASK(0)
#define VPU_HW_BTRS_MTL_VPU_D0I3_CONTROL 0x00000060u
#define VPU_HW_BTRS_MTL_VPU_D0I3_CONTROL_INPROGRESS_MASK BIT_MASK(0)
#define VPU_HW_BTRS_MTL_VPU_D0I3_CONTROL_I3_MASK BIT_MASK(2)
#define VPU_HW_BTRS_MTL_VPU_TELEMETRY_OFFSET 0x00000080u
#define VPU_HW_BTRS_MTL_VPU_TELEMETRY_SIZE 0x00000084u
#define VPU_HW_BTRS_MTL_VPU_TELEMETRY_ENABLE 0x00000088u
#define VPU_HW_BTRS_MTL_ATS_ERR_LOG_0 0x000000a0u
#define VPU_HW_BTRS_MTL_ATS_ERR_LOG_1 0x000000a4u
#define VPU_HW_BTRS_MTL_ATS_ERR_CLEAR 0x000000a8u
#define VPU_HW_BTRS_MTL_UFI_ERR_LOG 0x000000b0u
#define VPU_HW_BTRS_MTL_UFI_ERR_LOG_CQ_ID_MASK GENMASK(11, 0)
#define VPU_HW_BTRS_MTL_UFI_ERR_LOG_AXI_ID_MASK GENMASK(19, 12)
#define VPU_HW_BTRS_MTL_UFI_ERR_LOG_OPCODE_MASK GENMASK(24, 20)
#define VPU_HW_BTRS_MTL_UFI_ERR_CLEAR 0x000000b4u
#define VPU_HW_BTRS_LNL_INTERRUPT_STAT 0x00000000u
#define VPU_HW_BTRS_LNL_INTERRUPT_STAT_FREQ_CHANGE_MASK BIT_MASK(0)
#define VPU_HW_BTRS_LNL_INTERRUPT_STAT_ATS_ERR_MASK BIT_MASK(1)
@@ -79,12 +151,12 @@
#define VPU_HW_BTRS_LNL_TILE_FUSE 0x00000150u
#define VPU_HW_BTRS_LNL_TILE_FUSE_VALID_MASK BIT_MASK(0)
#define VPU_HW_BTRS_LNL_TILE_FUSE_CONFIG_MASK GENMASK(6, 1)
#define VPU_HW_BTRS_LNL_TILE_FUSE_CONFIG_MASK GENMASK(7, 1)
#define VPU_HW_BTRS_LNL_VPU_STATUS 0x00000154u
#define VPU_HW_BTRS_LNL_VPU_STATUS_READY_MASK BIT_MASK(0)
#define VPU_HW_BTRS_LNL_VPU_STATUS_IDLE_MASK BIT_MASK(1)
#define VPU_HW_BTRS_LNL_VPU_STATUS_DUP_IDLE_MASK BIT_MASK(2)
#define VPU_HW_BTRS_LNL_VPU_STATUS_DPU_IDLE_MASK BIT_MASK(2)
#define VPU_HW_BTRS_LNL_VPU_STATUS_CLOCK_RESOURCE_OWN_ACK_MASK BIT_MASK(6)
#define VPU_HW_BTRS_LNL_VPU_STATUS_POWER_RESOURCE_OWN_ACK_MASK BIT_MASK(7)
#define VPU_HW_BTRS_LNL_VPU_STATUS_PERF_CLK_MASK BIT_MASK(11)
@@ -109,4 +181,4 @@
#define VPU_HW_BTRS_LNL_FMAX_FUSE 0x00000178u
#define VPU_HW_BTRS_LNL_FMAX_FUSE_MAX_RATIO_MASK GENMASK(7, 0)
#endif /* __IVPU_HW_BTRS_LNL_REG_H__ */
#endif /* __IVPU_HW_BTRS_REG_H__ */
+113 -377
View File
@@ -7,62 +7,37 @@
#include "ivpu_fw.h"
#include "ivpu_gem.h"
#include "ivpu_hw.h"
#include "ivpu_hw_37xx_reg.h"
#include "ivpu_hw_40xx_reg.h"
#include "ivpu_hw_btrs.h"
#include "ivpu_hw_ip.h"
#include "ivpu_hw_ip_reg.h"
#include "ivpu_hw_reg_io.h"
#include "ivpu_mmu.h"
#include "ivpu_pm.h"
#define PWR_ISLAND_STATUS_TIMEOUT_US (5 * USEC_PER_MSEC)
#define TIM_SAFE_ENABLE 0xf1d0dead
#define TIM_WATCHDOG_RESET_VALUE 0xffffffff
#define ICB_0_IRQ_MASK_37XX ((REG_FLD(VPU_37XX_HOST_SS_ICB_STATUS_0, HOST_IPC_FIFO_INT)) | \
(REG_FLD(VPU_37XX_HOST_SS_ICB_STATUS_0, MMU_IRQ_0_INT)) | \
(REG_FLD(VPU_37XX_HOST_SS_ICB_STATUS_0, MMU_IRQ_1_INT)) | \
(REG_FLD(VPU_37XX_HOST_SS_ICB_STATUS_0, MMU_IRQ_2_INT)) | \
(REG_FLD(VPU_37XX_HOST_SS_ICB_STATUS_0, NOC_FIREWALL_INT)) | \
(REG_FLD(VPU_37XX_HOST_SS_ICB_STATUS_0, CPU_INT_REDIRECT_0_INT)) | \
(REG_FLD(VPU_37XX_HOST_SS_ICB_STATUS_0, CPU_INT_REDIRECT_1_INT)))
#define ICB_0_IRQ_MASK ((REG_FLD(VPU_37XX_HOST_SS_ICB_STATUS_0, HOST_IPC_FIFO_INT)) | \
(REG_FLD(VPU_37XX_HOST_SS_ICB_STATUS_0, MMU_IRQ_0_INT)) | \
(REG_FLD(VPU_37XX_HOST_SS_ICB_STATUS_0, MMU_IRQ_1_INT)) | \
(REG_FLD(VPU_37XX_HOST_SS_ICB_STATUS_0, MMU_IRQ_2_INT)) | \
(REG_FLD(VPU_37XX_HOST_SS_ICB_STATUS_0, NOC_FIREWALL_INT)) | \
(REG_FLD(VPU_37XX_HOST_SS_ICB_STATUS_0, CPU_INT_REDIRECT_0_INT)) | \
(REG_FLD(VPU_37XX_HOST_SS_ICB_STATUS_0, CPU_INT_REDIRECT_1_INT)))
#define ICB_1_IRQ_MASK_37XX ((REG_FLD(VPU_37XX_HOST_SS_ICB_STATUS_1, CPU_INT_REDIRECT_2_INT)) | \
(REG_FLD(VPU_37XX_HOST_SS_ICB_STATUS_1, CPU_INT_REDIRECT_3_INT)) | \
(REG_FLD(VPU_37XX_HOST_SS_ICB_STATUS_1, CPU_INT_REDIRECT_4_INT)))
#define ICB_1_IRQ_MASK ((REG_FLD(VPU_37XX_HOST_SS_ICB_STATUS_1, CPU_INT_REDIRECT_2_INT)) | \
(REG_FLD(VPU_37XX_HOST_SS_ICB_STATUS_1, CPU_INT_REDIRECT_3_INT)) | \
(REG_FLD(VPU_37XX_HOST_SS_ICB_STATUS_1, CPU_INT_REDIRECT_4_INT)))
#define ICB_0_1_IRQ_MASK_37XX ((((u64)ICB_1_IRQ_MASK_37XX) << 32) | ICB_0_IRQ_MASK_37XX)
#define ICB_0_1_IRQ_MASK ((((u64)ICB_1_IRQ_MASK) << 32) | ICB_0_IRQ_MASK)
#define ICB_0_IRQ_MASK_40XX ((REG_FLD(VPU_40XX_HOST_SS_ICB_STATUS_0, HOST_IPC_FIFO_INT)) | \
(REG_FLD(VPU_40XX_HOST_SS_ICB_STATUS_0, MMU_IRQ_0_INT)) | \
(REG_FLD(VPU_40XX_HOST_SS_ICB_STATUS_0, MMU_IRQ_1_INT)) | \
(REG_FLD(VPU_40XX_HOST_SS_ICB_STATUS_0, MMU_IRQ_2_INT)) | \
(REG_FLD(VPU_40XX_HOST_SS_ICB_STATUS_0, NOC_FIREWALL_INT)) | \
(REG_FLD(VPU_40XX_HOST_SS_ICB_STATUS_0, CPU_INT_REDIRECT_0_INT)) | \
(REG_FLD(VPU_40XX_HOST_SS_ICB_STATUS_0, CPU_INT_REDIRECT_1_INT)))
#define ICB_1_IRQ_MASK_40XX ((REG_FLD(VPU_40XX_HOST_SS_ICB_STATUS_1, CPU_INT_REDIRECT_2_INT)) | \
(REG_FLD(VPU_40XX_HOST_SS_ICB_STATUS_1, CPU_INT_REDIRECT_3_INT)) | \
(REG_FLD(VPU_40XX_HOST_SS_ICB_STATUS_1, CPU_INT_REDIRECT_4_INT)))
#define ICB_0_1_IRQ_MASK_40XX ((((u64)ICB_1_IRQ_MASK_40XX) << 32) | ICB_0_IRQ_MASK_40XX)
#define ITF_FIREWALL_VIOLATION_MASK_37XX ((REG_FLD(VPU_37XX_HOST_SS_FW_SOC_IRQ_EN, CSS_ROM_CMX)) | \
(REG_FLD(VPU_37XX_HOST_SS_FW_SOC_IRQ_EN, CSS_DBG)) | \
(REG_FLD(VPU_37XX_HOST_SS_FW_SOC_IRQ_EN, CSS_CTRL)) | \
(REG_FLD(VPU_37XX_HOST_SS_FW_SOC_IRQ_EN, DEC400)) | \
(REG_FLD(VPU_37XX_HOST_SS_FW_SOC_IRQ_EN, MSS_NCE)) | \
(REG_FLD(VPU_37XX_HOST_SS_FW_SOC_IRQ_EN, MSS_MBI)) | \
(REG_FLD(VPU_37XX_HOST_SS_FW_SOC_IRQ_EN, MSS_MBI_CMX)))
#define ITF_FIREWALL_VIOLATION_MASK_40XX ((REG_FLD(VPU_40XX_HOST_SS_FW_SOC_IRQ_EN, CSS_ROM_CMX)) | \
(REG_FLD(VPU_40XX_HOST_SS_FW_SOC_IRQ_EN, CSS_DBG)) | \
(REG_FLD(VPU_40XX_HOST_SS_FW_SOC_IRQ_EN, CSS_CTRL)) | \
(REG_FLD(VPU_40XX_HOST_SS_FW_SOC_IRQ_EN, DEC400)) | \
(REG_FLD(VPU_40XX_HOST_SS_FW_SOC_IRQ_EN, MSS_NCE)) | \
(REG_FLD(VPU_40XX_HOST_SS_FW_SOC_IRQ_EN, MSS_MBI)) | \
(REG_FLD(VPU_40XX_HOST_SS_FW_SOC_IRQ_EN, MSS_MBI_CMX)))
#define ITF_FIREWALL_VIOLATION_MASK ((REG_FLD(VPU_37XX_HOST_SS_FW_SOC_IRQ_EN, CSS_ROM_CMX)) | \
(REG_FLD(VPU_37XX_HOST_SS_FW_SOC_IRQ_EN, CSS_DBG)) | \
(REG_FLD(VPU_37XX_HOST_SS_FW_SOC_IRQ_EN, CSS_CTRL)) | \
(REG_FLD(VPU_37XX_HOST_SS_FW_SOC_IRQ_EN, DEC400)) | \
(REG_FLD(VPU_37XX_HOST_SS_FW_SOC_IRQ_EN, MSS_NCE)) | \
(REG_FLD(VPU_37XX_HOST_SS_FW_SOC_IRQ_EN, MSS_MBI)) | \
(REG_FLD(VPU_37XX_HOST_SS_FW_SOC_IRQ_EN, MSS_MBI_CMX)))
static int wait_for_ip_bar(struct ivpu_device *vdev)
{
@@ -80,7 +55,7 @@ static void host_ss_rst_clr(struct ivpu_device *vdev)
REGV_WR32(VPU_37XX_HOST_SS_CPR_RST_CLR, val);
}
static int host_ss_noc_qreqn_check_37xx(struct ivpu_device *vdev, u32 exp_val)
static int host_ss_noc_qreqn_check(struct ivpu_device *vdev, u32 exp_val)
{
u32 val = REGV_RD32(VPU_37XX_HOST_SS_NOC_QREQN);
@@ -90,25 +65,7 @@ static int host_ss_noc_qreqn_check_37xx(struct ivpu_device *vdev, u32 exp_val)
return 0;
}
static int host_ss_noc_qreqn_check_40xx(struct ivpu_device *vdev, u32 exp_val)
{
u32 val = REGV_RD32(VPU_40XX_HOST_SS_NOC_QREQN);
if (!REG_TEST_FLD_NUM(VPU_40XX_HOST_SS_NOC_QREQN, TOP_SOCMMIO, exp_val, val))
return -EIO;
return 0;
}
static int host_ss_noc_qreqn_check(struct ivpu_device *vdev, u32 exp_val)
{
if (ivpu_hw_ip_gen(vdev) == IVPU_HW_IP_37XX)
return host_ss_noc_qreqn_check_37xx(vdev, exp_val);
else
return host_ss_noc_qreqn_check_40xx(vdev, exp_val);
}
static int host_ss_noc_qacceptn_check_37xx(struct ivpu_device *vdev, u32 exp_val)
static int host_ss_noc_qacceptn_check(struct ivpu_device *vdev, u32 exp_val)
{
u32 val = REGV_RD32(VPU_37XX_HOST_SS_NOC_QACCEPTN);
@@ -118,25 +75,7 @@ static int host_ss_noc_qacceptn_check_37xx(struct ivpu_device *vdev, u32 exp_val
return 0;
}
static int host_ss_noc_qacceptn_check_40xx(struct ivpu_device *vdev, u32 exp_val)
{
u32 val = REGV_RD32(VPU_40XX_HOST_SS_NOC_QACCEPTN);
if (!REG_TEST_FLD_NUM(VPU_40XX_HOST_SS_NOC_QACCEPTN, TOP_SOCMMIO, exp_val, val))
return -EIO;
return 0;
}
static int host_ss_noc_qacceptn_check(struct ivpu_device *vdev, u32 exp_val)
{
if (ivpu_hw_ip_gen(vdev) == IVPU_HW_IP_37XX)
return host_ss_noc_qacceptn_check_37xx(vdev, exp_val);
else
return host_ss_noc_qacceptn_check_40xx(vdev, exp_val);
}
static int host_ss_noc_qdeny_check_37xx(struct ivpu_device *vdev, u32 exp_val)
static int host_ss_noc_qdeny_check(struct ivpu_device *vdev, u32 exp_val)
{
u32 val = REGV_RD32(VPU_37XX_HOST_SS_NOC_QDENY);
@@ -146,24 +85,6 @@ static int host_ss_noc_qdeny_check_37xx(struct ivpu_device *vdev, u32 exp_val)
return 0;
}
static int host_ss_noc_qdeny_check_40xx(struct ivpu_device *vdev, u32 exp_val)
{
u32 val = REGV_RD32(VPU_40XX_HOST_SS_NOC_QDENY);
if (!REG_TEST_FLD_NUM(VPU_40XX_HOST_SS_NOC_QDENY, TOP_SOCMMIO, exp_val, val))
return -EIO;
return 0;
}
static int host_ss_noc_qdeny_check(struct ivpu_device *vdev, u32 exp_val)
{
if (ivpu_hw_ip_gen(vdev) == IVPU_HW_IP_37XX)
return host_ss_noc_qdeny_check_37xx(vdev, exp_val);
else
return host_ss_noc_qdeny_check_40xx(vdev, exp_val);
}
static int top_noc_qrenqn_check_37xx(struct ivpu_device *vdev, u32 exp_val)
{
u32 val = REGV_RD32(VPU_37XX_TOP_NOC_QREQN);
@@ -177,9 +98,9 @@ static int top_noc_qrenqn_check_37xx(struct ivpu_device *vdev, u32 exp_val)
static int top_noc_qrenqn_check_40xx(struct ivpu_device *vdev, u32 exp_val)
{
u32 val = REGV_RD32(VPU_40XX_TOP_NOC_QREQN);
u32 val = REGV_RD32(VPU_37XX_TOP_NOC_QREQN);
if (!REG_TEST_FLD_NUM(VPU_40XX_TOP_NOC_QREQN, CPU_CTRL, exp_val, val) ||
if (!REG_TEST_FLD_NUM(VPU_37XX_TOP_NOC_QREQN, CPU_CTRL, exp_val, val) ||
!REG_TEST_FLD_NUM(VPU_40XX_TOP_NOC_QREQN, HOSTIF_L2CACHE, exp_val, val))
return -EIO;
@@ -226,7 +147,7 @@ int ivpu_hw_ip_host_ss_configure(struct ivpu_device *vdev)
return ret;
}
static void idle_gen_drive_37xx(struct ivpu_device *vdev, bool enable)
static void idle_gen_drive(struct ivpu_device *vdev, bool enable)
{
u32 val = REGV_RD32(VPU_37XX_HOST_SS_AON_VPU_IDLE_GEN);
@@ -238,51 +159,47 @@ static void idle_gen_drive_37xx(struct ivpu_device *vdev, bool enable)
REGV_WR32(VPU_37XX_HOST_SS_AON_VPU_IDLE_GEN, val);
}
static void idle_gen_drive_40xx(struct ivpu_device *vdev, bool enable)
{
u32 val = REGV_RD32(VPU_40XX_HOST_SS_AON_IDLE_GEN);
if (enable)
val = REG_SET_FLD(VPU_40XX_HOST_SS_AON_IDLE_GEN, EN, val);
else
val = REG_CLR_FLD(VPU_40XX_HOST_SS_AON_IDLE_GEN, EN, val);
REGV_WR32(VPU_40XX_HOST_SS_AON_IDLE_GEN, val);
}
void ivpu_hw_ip_idle_gen_enable(struct ivpu_device *vdev)
{
if (ivpu_hw_ip_gen(vdev) == IVPU_HW_IP_37XX)
idle_gen_drive_37xx(vdev, true);
else
idle_gen_drive_40xx(vdev, true);
idle_gen_drive(vdev, true);
}
void ivpu_hw_ip_idle_gen_disable(struct ivpu_device *vdev)
{
if (ivpu_hw_ip_gen(vdev) == IVPU_HW_IP_37XX)
idle_gen_drive_37xx(vdev, false);
else
idle_gen_drive_40xx(vdev, false);
idle_gen_drive(vdev, false);
}
static void
pwr_island_delay_set_50xx(struct ivpu_device *vdev, u32 post, u32 post1, u32 post2, u32 status)
pwr_island_delay_write(struct ivpu_device *vdev, u32 post, u32 post1, u32 post2, u32 status)
{
u32 val;
val = REGV_RD32(VPU_50XX_HOST_SS_AON_PWR_ISLAND_EN_POST_DLY);
val = REG_SET_FLD_NUM(VPU_50XX_HOST_SS_AON_PWR_ISLAND_EN_POST_DLY, POST_DLY, post, val);
val = REG_SET_FLD_NUM(VPU_50XX_HOST_SS_AON_PWR_ISLAND_EN_POST_DLY, POST_DLY, post, 0);
val = REG_SET_FLD_NUM(VPU_50XX_HOST_SS_AON_PWR_ISLAND_EN_POST_DLY, POST1_DLY, post1, val);
val = REG_SET_FLD_NUM(VPU_50XX_HOST_SS_AON_PWR_ISLAND_EN_POST_DLY, POST2_DLY, post2, val);
REGV_WR32(VPU_50XX_HOST_SS_AON_PWR_ISLAND_EN_POST_DLY, val);
val = REGV_RD32(VPU_50XX_HOST_SS_AON_PWR_ISLAND_STATUS_DLY);
val = REG_SET_FLD_NUM(VPU_50XX_HOST_SS_AON_PWR_ISLAND_STATUS_DLY, STATUS_DLY, status, val);
val = REG_SET_FLD_NUM(VPU_50XX_HOST_SS_AON_PWR_ISLAND_STATUS_DLY, STATUS_DLY, status, 0);
REGV_WR32(VPU_50XX_HOST_SS_AON_PWR_ISLAND_STATUS_DLY, val);
}
static void pwr_island_trickle_drive_37xx(struct ivpu_device *vdev, bool enable)
static void pwr_island_delay_set_50xx(struct ivpu_device *vdev)
{
if (vdev->hw->pll.profiling_freq == PLL_PROFILING_FREQ_HIGH)
pwr_island_delay_write(vdev, 18, 0, 0, 46);
else
pwr_island_delay_write(vdev, 0, 0, 0, 3);
}
static void pwr_island_delay_set_60xx(struct ivpu_device *vdev)
{
if (vdev->hw->pll.profiling_freq == PLL_PROFILING_FREQ_HIGH)
pwr_island_delay_write(vdev, 198, 0, 198, 0);
else
pwr_island_delay_write(vdev, 17, 0, 17, 0);
}
static void pwr_island_trickle_drive(struct ivpu_device *vdev, bool enable)
{
u32 val = REGV_RD32(VPU_37XX_HOST_SS_AON_PWR_ISLAND_TRICKLE_EN0);
@@ -294,19 +211,7 @@ static void pwr_island_trickle_drive_37xx(struct ivpu_device *vdev, bool enable)
REGV_WR32(VPU_37XX_HOST_SS_AON_PWR_ISLAND_TRICKLE_EN0, val);
}
static void pwr_island_trickle_drive_40xx(struct ivpu_device *vdev, bool enable)
{
u32 val = REGV_RD32(VPU_40XX_HOST_SS_AON_PWR_ISLAND_TRICKLE_EN0);
if (enable)
val = REG_SET_FLD(VPU_40XX_HOST_SS_AON_PWR_ISLAND_TRICKLE_EN0, CSS_CPU, val);
else
val = REG_CLR_FLD(VPU_40XX_HOST_SS_AON_PWR_ISLAND_TRICKLE_EN0, CSS_CPU, val);
REGV_WR32(VPU_40XX_HOST_SS_AON_PWR_ISLAND_TRICKLE_EN0, val);
}
static void pwr_island_drive_37xx(struct ivpu_device *vdev, bool enable)
static void pwr_island_drive(struct ivpu_device *vdev, bool enable)
{
u32 val = REGV_RD32(VPU_37XX_HOST_SS_AON_PWR_ISLAND_EN0);
@@ -318,29 +223,11 @@ static void pwr_island_drive_37xx(struct ivpu_device *vdev, bool enable)
REGV_WR32(VPU_37XX_HOST_SS_AON_PWR_ISLAND_EN0, val);
}
static void pwr_island_drive_40xx(struct ivpu_device *vdev, bool enable)
{
u32 val = REGV_RD32(VPU_40XX_HOST_SS_AON_PWR_ISLAND_EN0);
if (enable)
val = REG_SET_FLD(VPU_40XX_HOST_SS_AON_PWR_ISLAND_EN0, CSS_CPU, val);
else
val = REG_CLR_FLD(VPU_40XX_HOST_SS_AON_PWR_ISLAND_EN0, CSS_CPU, val);
REGV_WR32(VPU_40XX_HOST_SS_AON_PWR_ISLAND_EN0, val);
}
static void pwr_island_enable(struct ivpu_device *vdev)
{
if (ivpu_hw_ip_gen(vdev) == IVPU_HW_IP_37XX) {
pwr_island_trickle_drive_37xx(vdev, true);
ndelay(500);
pwr_island_drive_37xx(vdev, true);
} else {
pwr_island_trickle_drive_40xx(vdev, true);
ndelay(500);
pwr_island_drive_40xx(vdev, true);
}
pwr_island_trickle_drive(vdev, true);
ndelay(500);
pwr_island_drive(vdev, true);
}
static int wait_for_pwr_island_status(struct ivpu_device *vdev, u32 exp_val)
@@ -348,15 +235,11 @@ static int wait_for_pwr_island_status(struct ivpu_device *vdev, u32 exp_val)
if (IVPU_WA(punit_disabled))
return 0;
if (ivpu_hw_ip_gen(vdev) == IVPU_HW_IP_37XX)
return REGV_POLL_FLD(VPU_37XX_HOST_SS_AON_PWR_ISLAND_STATUS0, MSS_CPU, exp_val,
PWR_ISLAND_STATUS_TIMEOUT_US);
else
return REGV_POLL_FLD(VPU_40XX_HOST_SS_AON_PWR_ISLAND_STATUS0, CSS_CPU, exp_val,
PWR_ISLAND_STATUS_TIMEOUT_US);
return REGV_POLL_FLD(VPU_37XX_HOST_SS_AON_PWR_ISLAND_STATUS0, MSS_CPU, exp_val,
vdev->timeout.pwr_island_status);
}
static void pwr_island_isolation_drive_37xx(struct ivpu_device *vdev, bool enable)
static void pwr_island_isolation_drive(struct ivpu_device *vdev, bool enable)
{
u32 val = REGV_RD32(VPU_37XX_HOST_SS_AON_PWR_ISO_EN0);
@@ -368,26 +251,6 @@ static void pwr_island_isolation_drive_37xx(struct ivpu_device *vdev, bool enabl
REGV_WR32(VPU_37XX_HOST_SS_AON_PWR_ISO_EN0, val);
}
static void pwr_island_isolation_drive_40xx(struct ivpu_device *vdev, bool enable)
{
u32 val = REGV_RD32(VPU_40XX_HOST_SS_AON_PWR_ISO_EN0);
if (enable)
val = REG_SET_FLD(VPU_40XX_HOST_SS_AON_PWR_ISO_EN0, CSS_CPU, val);
else
val = REG_CLR_FLD(VPU_40XX_HOST_SS_AON_PWR_ISO_EN0, CSS_CPU, val);
REGV_WR32(VPU_40XX_HOST_SS_AON_PWR_ISO_EN0, val);
}
static void pwr_island_isolation_drive(struct ivpu_device *vdev, bool enable)
{
if (ivpu_hw_ip_gen(vdev) == IVPU_HW_IP_37XX)
pwr_island_isolation_drive_37xx(vdev, enable);
else
pwr_island_isolation_drive_40xx(vdev, enable);
}
static void pwr_island_isolation_disable(struct ivpu_device *vdev)
{
pwr_island_isolation_drive(vdev, false);
@@ -487,7 +350,7 @@ static void host_ss_rst_enable(struct ivpu_device *vdev)
host_ss_rst_drive(vdev, true);
}
static void host_ss_noc_qreqn_top_socmmio_drive_37xx(struct ivpu_device *vdev, bool enable)
static void host_ss_noc_qreqn_top_socmmio_drive(struct ivpu_device *vdev, bool enable)
{
u32 val = REGV_RD32(VPU_37XX_HOST_SS_NOC_QREQN);
@@ -498,25 +361,6 @@ static void host_ss_noc_qreqn_top_socmmio_drive_37xx(struct ivpu_device *vdev, b
REGV_WR32(VPU_37XX_HOST_SS_NOC_QREQN, val);
}
static void host_ss_noc_qreqn_top_socmmio_drive_40xx(struct ivpu_device *vdev, bool enable)
{
u32 val = REGV_RD32(VPU_40XX_HOST_SS_NOC_QREQN);
if (enable)
val = REG_SET_FLD(VPU_40XX_HOST_SS_NOC_QREQN, TOP_SOCMMIO, val);
else
val = REG_CLR_FLD(VPU_40XX_HOST_SS_NOC_QREQN, TOP_SOCMMIO, val);
REGV_WR32(VPU_40XX_HOST_SS_NOC_QREQN, val);
}
static void host_ss_noc_qreqn_top_socmmio_drive(struct ivpu_device *vdev, bool enable)
{
if (ivpu_hw_ip_gen(vdev) == IVPU_HW_IP_37XX)
host_ss_noc_qreqn_top_socmmio_drive_37xx(vdev, enable);
else
host_ss_noc_qreqn_top_socmmio_drive_40xx(vdev, enable);
}
static int host_ss_axi_drive(struct ivpu_device *vdev, bool enable)
{
int ret;
@@ -538,17 +382,17 @@ static int host_ss_axi_drive(struct ivpu_device *vdev, bool enable)
static void top_noc_qreqn_drive_40xx(struct ivpu_device *vdev, bool enable)
{
u32 val = REGV_RD32(VPU_40XX_TOP_NOC_QREQN);
u32 val = REGV_RD32(VPU_37XX_TOP_NOC_QREQN);
if (enable) {
val = REG_SET_FLD(VPU_40XX_TOP_NOC_QREQN, CPU_CTRL, val);
val = REG_SET_FLD(VPU_37XX_TOP_NOC_QREQN, CPU_CTRL, val);
val = REG_SET_FLD(VPU_40XX_TOP_NOC_QREQN, HOSTIF_L2CACHE, val);
} else {
val = REG_CLR_FLD(VPU_40XX_TOP_NOC_QREQN, CPU_CTRL, val);
val = REG_CLR_FLD(VPU_37XX_TOP_NOC_QREQN, CPU_CTRL, val);
val = REG_CLR_FLD(VPU_40XX_TOP_NOC_QREQN, HOSTIF_L2CACHE, val);
}
REGV_WR32(VPU_40XX_TOP_NOC_QREQN, val);
REGV_WR32(VPU_37XX_TOP_NOC_QREQN, val);
}
static void top_noc_qreqn_drive_37xx(struct ivpu_device *vdev, bool enable)
@@ -592,9 +436,9 @@ static int top_noc_qacceptn_check_37xx(struct ivpu_device *vdev, u32 exp_val)
static int top_noc_qacceptn_check_40xx(struct ivpu_device *vdev, u32 exp_val)
{
u32 val = REGV_RD32(VPU_40XX_TOP_NOC_QACCEPTN);
u32 val = REGV_RD32(VPU_37XX_TOP_NOC_QACCEPTN);
if (!REG_TEST_FLD_NUM(VPU_40XX_TOP_NOC_QACCEPTN, CPU_CTRL, exp_val, val) ||
if (!REG_TEST_FLD_NUM(VPU_37XX_TOP_NOC_QACCEPTN, CPU_CTRL, exp_val, val) ||
!REG_TEST_FLD_NUM(VPU_40XX_TOP_NOC_QACCEPTN, HOSTIF_L2CACHE, exp_val, val))
return -EIO;
@@ -622,9 +466,9 @@ static int top_noc_qdeny_check_37xx(struct ivpu_device *vdev, u32 exp_val)
static int top_noc_qdeny_check_40xx(struct ivpu_device *vdev, u32 exp_val)
{
u32 val = REGV_RD32(VPU_40XX_TOP_NOC_QDENY);
u32 val = REGV_RD32(VPU_37XX_TOP_NOC_QDENY);
if (!REG_TEST_FLD_NUM(VPU_40XX_TOP_NOC_QDENY, CPU_CTRL, exp_val, val) ||
if (!REG_TEST_FLD_NUM(VPU_37XX_TOP_NOC_QDENY, CPU_CTRL, exp_val, val) ||
!REG_TEST_FLD_NUM(VPU_40XX_TOP_NOC_QDENY, HOSTIF_L2CACHE, exp_val, val))
return -EIO;
@@ -677,35 +521,19 @@ static void dpu_active_drive_37xx(struct ivpu_device *vdev, bool enable)
static void pwr_island_delay_set(struct ivpu_device *vdev)
{
bool high = vdev->hw->pll.profiling_freq == PLL_PROFILING_FREQ_HIGH;
u32 post, post1, post2, status;
if (ivpu_hw_ip_gen(vdev) < IVPU_HW_IP_50XX)
switch (ivpu_hw_ip_gen(vdev)) {
case IVPU_HW_IP_37XX:
case IVPU_HW_IP_40XX:
return;
switch (ivpu_device_id(vdev)) {
case PCI_DEVICE_ID_WCL:
case PCI_DEVICE_ID_PTL_P:
post = high ? 18 : 0;
post1 = 0;
post2 = 0;
status = high ? 46 : 3;
break;
case IVPU_HW_IP_50XX:
pwr_island_delay_set_50xx(vdev);
return;
case PCI_DEVICE_ID_NVL:
post = high ? 198 : 17;
post1 = 0;
post2 = high ? 198 : 17;
status = 0;
break;
default:
dump_stack();
ivpu_err(vdev, "Unknown device ID\n");
case IVPU_HW_IP_60XX:
pwr_island_delay_set_60xx(vdev);
return;
}
pwr_island_delay_set_50xx(vdev, post, post1, post2, status);
}
int ivpu_hw_ip_pwr_domain_enable(struct ivpu_device *vdev)
@@ -762,7 +590,7 @@ static void ivpu_hw_ip_snoop_disable_37xx(struct ivpu_device *vdev)
static void ivpu_hw_ip_snoop_disable_40xx(struct ivpu_device *vdev)
{
u32 val = REGV_RD32(VPU_40XX_HOST_IF_TCU_PTW_OVERRIDES);
u32 val = REGV_RD32(VPU_37XX_HOST_IF_TCU_PTW_OVERRIDES);
val = REG_SET_FLD(VPU_40XX_HOST_IF_TCU_PTW_OVERRIDES, SNOOP_OVERRIDE_EN, val);
val = REG_SET_FLD(VPU_40XX_HOST_IF_TCU_PTW_OVERRIDES, AW_SNOOP_OVERRIDE, val);
@@ -772,7 +600,7 @@ static void ivpu_hw_ip_snoop_disable_40xx(struct ivpu_device *vdev)
else
val = REG_CLR_FLD(VPU_40XX_HOST_IF_TCU_PTW_OVERRIDES, AR_SNOOP_OVERRIDE, val);
REGV_WR32(VPU_40XX_HOST_IF_TCU_PTW_OVERRIDES, val);
REGV_WR32(VPU_37XX_HOST_IF_TCU_PTW_OVERRIDES, val);
}
void ivpu_hw_ip_snoop_disable(struct ivpu_device *vdev)
@@ -797,24 +625,29 @@ static void ivpu_hw_ip_tbu_mmu_enable_37xx(struct ivpu_device *vdev)
static void ivpu_hw_ip_tbu_mmu_enable_40xx(struct ivpu_device *vdev)
{
u32 val = REGV_RD32(VPU_40XX_HOST_IF_TBU_MMUSSIDV);
u32 val = REGV_RD32(VPU_37XX_HOST_IF_TBU_MMUSSIDV);
val = REG_SET_FLD(VPU_40XX_HOST_IF_TBU_MMUSSIDV, TBU0_AWMMUSSIDV, val);
val = REG_SET_FLD(VPU_40XX_HOST_IF_TBU_MMUSSIDV, TBU0_ARMMUSSIDV, val);
val = REG_SET_FLD(VPU_40XX_HOST_IF_TBU_MMUSSIDV, TBU1_AWMMUSSIDV, val);
val = REG_SET_FLD(VPU_40XX_HOST_IF_TBU_MMUSSIDV, TBU1_ARMMUSSIDV, val);
val = REG_SET_FLD(VPU_40XX_HOST_IF_TBU_MMUSSIDV, TBU2_AWMMUSSIDV, val);
val = REG_SET_FLD(VPU_40XX_HOST_IF_TBU_MMUSSIDV, TBU2_ARMMUSSIDV, val);
val = REG_SET_FLD(VPU_37XX_HOST_IF_TBU_MMUSSIDV, TBU0_AWMMUSSIDV, val);
val = REG_SET_FLD(VPU_37XX_HOST_IF_TBU_MMUSSIDV, TBU0_ARMMUSSIDV, val);
val = REG_SET_FLD(VPU_37XX_HOST_IF_TBU_MMUSSIDV, TBU1_AWMMUSSIDV, val);
val = REG_SET_FLD(VPU_37XX_HOST_IF_TBU_MMUSSIDV, TBU1_ARMMUSSIDV, val);
val = REG_SET_FLD(VPU_37XX_HOST_IF_TBU_MMUSSIDV, TBU2_AWMMUSSIDV, val);
val = REG_SET_FLD(VPU_37XX_HOST_IF_TBU_MMUSSIDV, TBU2_ARMMUSSIDV, val);
REGV_WR32(VPU_40XX_HOST_IF_TBU_MMUSSIDV, val);
REGV_WR32(VPU_37XX_HOST_IF_TBU_MMUSSIDV, val);
}
void ivpu_hw_ip_tbu_mmu_enable(struct ivpu_device *vdev)
{
if (ivpu_hw_ip_gen(vdev) == IVPU_HW_IP_37XX)
switch (ivpu_hw_ip_gen(vdev)) {
case IVPU_HW_IP_37XX:
return ivpu_hw_ip_tbu_mmu_enable_37xx(vdev);
else
case IVPU_HW_IP_40XX:
case IVPU_HW_IP_50XX:
case IVPU_HW_IP_60XX:
return ivpu_hw_ip_tbu_mmu_enable_40xx(vdev);
}
}
static inline u64 get_entry_point_addr(struct ivpu_device *vdev)
@@ -938,29 +771,22 @@ static int soc_cpu_boot_60xx(struct ivpu_device *vdev)
int ivpu_hw_ip_soc_cpu_boot(struct ivpu_device *vdev)
{
int ret;
switch (ivpu_hw_ip_gen(vdev)) {
case IVPU_HW_IP_37XX:
ret = soc_cpu_boot_37xx(vdev);
break;
case IVPU_HW_IP_40XX:
case IVPU_HW_IP_50XX:
ret = soc_cpu_boot_40xx(vdev);
break;
default:
ret = soc_cpu_boot_60xx(vdev);
}
if (ret)
return ret;
ivpu_dbg(vdev, PM, "Booting firmware, mode: %s\n",
ivpu_fw_is_warm_boot(vdev) ? "warm boot" : "cold boot");
return 0;
switch (ivpu_hw_ip_gen(vdev)) {
case IVPU_HW_IP_37XX:
return soc_cpu_boot_37xx(vdev);
case IVPU_HW_IP_40XX:
case IVPU_HW_IP_50XX:
return soc_cpu_boot_40xx(vdev);
case IVPU_HW_IP_60XX:
return soc_cpu_boot_60xx(vdev);
}
return -EINVAL;
}
static void wdt_disable_37xx(struct ivpu_device *vdev)
@@ -1004,55 +830,30 @@ void ivpu_hw_ip_wdt_disable(struct ivpu_device *vdev)
return wdt_disable_40xx(vdev);
}
static u32 ipc_rx_count_get_37xx(struct ivpu_device *vdev)
u32 ivpu_hw_ip_ipc_rx_count_get(struct ivpu_device *vdev)
{
u32 count = readl(vdev->regv + VPU_37XX_HOST_SS_TIM_IPC_FIFO_STAT);
return REG_GET_FLD(VPU_37XX_HOST_SS_TIM_IPC_FIFO_STAT, FILL_LEVEL, count);
}
static u32 ipc_rx_count_get_40xx(struct ivpu_device *vdev)
{
u32 count = readl(vdev->regv + VPU_40XX_HOST_SS_TIM_IPC_FIFO_STAT);
return REG_GET_FLD(VPU_40XX_HOST_SS_TIM_IPC_FIFO_STAT, FILL_LEVEL, count);
}
u32 ivpu_hw_ip_ipc_rx_count_get(struct ivpu_device *vdev)
{
if (ivpu_hw_ip_gen(vdev) == IVPU_HW_IP_37XX)
return ipc_rx_count_get_37xx(vdev);
else
return ipc_rx_count_get_40xx(vdev);
}
void ivpu_hw_ip_irq_enable(struct ivpu_device *vdev)
{
if (ivpu_hw_ip_gen(vdev) == IVPU_HW_IP_37XX) {
REGV_WR32(VPU_37XX_HOST_SS_FW_SOC_IRQ_EN, ITF_FIREWALL_VIOLATION_MASK_37XX);
REGV_WR64(VPU_37XX_HOST_SS_ICB_ENABLE_0, ICB_0_1_IRQ_MASK_37XX);
} else {
REGV_WR32(VPU_40XX_HOST_SS_FW_SOC_IRQ_EN, ITF_FIREWALL_VIOLATION_MASK_40XX);
REGV_WR64(VPU_40XX_HOST_SS_ICB_ENABLE_0, ICB_0_1_IRQ_MASK_40XX);
}
REGV_WR32(VPU_37XX_HOST_SS_FW_SOC_IRQ_EN, ITF_FIREWALL_VIOLATION_MASK);
REGV_WR64(VPU_37XX_HOST_SS_ICB_ENABLE_0, ICB_0_1_IRQ_MASK);
}
void ivpu_hw_ip_irq_disable(struct ivpu_device *vdev)
{
if (ivpu_hw_ip_gen(vdev) == IVPU_HW_IP_37XX) {
REGV_WR64(VPU_37XX_HOST_SS_ICB_ENABLE_0, 0x0ull);
REGV_WR32(VPU_37XX_HOST_SS_FW_SOC_IRQ_EN, 0x0);
} else {
REGV_WR64(VPU_40XX_HOST_SS_ICB_ENABLE_0, 0x0ull);
REGV_WR32(VPU_40XX_HOST_SS_FW_SOC_IRQ_EN, 0x0ul);
}
REGV_WR64(VPU_37XX_HOST_SS_ICB_ENABLE_0, 0x0ull);
REGV_WR32(VPU_37XX_HOST_SS_FW_SOC_IRQ_EN, 0x0);
}
static void diagnose_failure_37xx(struct ivpu_device *vdev)
void ivpu_hw_ip_diagnose_failure(struct ivpu_device *vdev)
{
u32 reg = REGV_RD32(VPU_37XX_HOST_SS_ICB_STATUS_0) & ICB_0_IRQ_MASK_37XX;
u32 reg = REGV_RD32(VPU_37XX_HOST_SS_ICB_STATUS_0) & ICB_0_IRQ_MASK;
if (ipc_rx_count_get_37xx(vdev))
if (ivpu_hw_ip_ipc_rx_count_get(vdev))
ivpu_err(vdev, "IPC FIFO queue not empty, missed IPC IRQ");
if (REG_TEST_FLD(VPU_37XX_HOST_SS_ICB_STATUS_0, CPU_INT_REDIRECT_0_INT, reg))
@@ -1065,37 +866,9 @@ static void diagnose_failure_37xx(struct ivpu_device *vdev)
ivpu_err(vdev, "NOC Firewall irq detected\n");
}
static void diagnose_failure_40xx(struct ivpu_device *vdev)
{
u32 reg = REGV_RD32(VPU_40XX_HOST_SS_ICB_STATUS_0) & ICB_0_IRQ_MASK_40XX;
if (ipc_rx_count_get_40xx(vdev))
ivpu_err(vdev, "IPC FIFO queue not empty, missed IPC IRQ");
if (REG_TEST_FLD(VPU_40XX_HOST_SS_ICB_STATUS_0, CPU_INT_REDIRECT_0_INT, reg))
ivpu_err(vdev, "WDT MSS timeout detected\n");
if (REG_TEST_FLD(VPU_40XX_HOST_SS_ICB_STATUS_0, CPU_INT_REDIRECT_1_INT, reg))
ivpu_err(vdev, "WDT NCE timeout detected\n");
if (REG_TEST_FLD(VPU_40XX_HOST_SS_ICB_STATUS_0, NOC_FIREWALL_INT, reg))
ivpu_err(vdev, "NOC Firewall irq detected\n");
}
void ivpu_hw_ip_diagnose_failure(struct ivpu_device *vdev)
{
if (ivpu_hw_ip_gen(vdev) == IVPU_HW_IP_37XX)
diagnose_failure_37xx(vdev);
else
diagnose_failure_40xx(vdev);
}
void ivpu_hw_ip_irq_clear(struct ivpu_device *vdev)
{
if (ivpu_hw_ip_gen(vdev) == IVPU_HW_IP_37XX)
REGV_WR64(VPU_37XX_HOST_SS_ICB_CLEAR_0, ICB_0_1_IRQ_MASK_37XX);
else
REGV_WR64(VPU_40XX_HOST_SS_ICB_CLEAR_0, ICB_0_1_IRQ_MASK_40XX);
REGV_WR64(VPU_37XX_HOST_SS_ICB_CLEAR_0, ICB_0_1_IRQ_MASK);
}
static void irq_wdt_nce_handler(struct ivpu_device *vdev)
@@ -1118,9 +891,9 @@ static void irq_noc_firewall_handler(struct ivpu_device *vdev)
}
/* Handler for IRQs from NPU core */
bool ivpu_hw_ip_irq_handler_37xx(struct ivpu_device *vdev, int irq)
bool ivpu_hw_ip_irq_handler(struct ivpu_device *vdev, int irq)
{
u32 status = REGV_RD32(VPU_37XX_HOST_SS_ICB_STATUS_0) & ICB_0_IRQ_MASK_37XX;
u32 status = REGV_RD32(VPU_37XX_HOST_SS_ICB_STATUS_0) & ICB_0_IRQ_MASK;
if (!status)
return false;
@@ -1151,40 +924,6 @@ bool ivpu_hw_ip_irq_handler_37xx(struct ivpu_device *vdev, int irq)
return true;
}
/* Handler for IRQs from NPU core */
bool ivpu_hw_ip_irq_handler_40xx(struct ivpu_device *vdev, int irq)
{
u32 status = REGV_RD32(VPU_40XX_HOST_SS_ICB_STATUS_0) & ICB_0_IRQ_MASK_40XX;
if (!status)
return false;
REGV_WR32(VPU_40XX_HOST_SS_ICB_CLEAR_0, status);
if (REG_TEST_FLD(VPU_40XX_HOST_SS_ICB_STATUS_0, MMU_IRQ_0_INT, status))
ivpu_mmu_irq_evtq_handler(vdev);
if (REG_TEST_FLD(VPU_40XX_HOST_SS_ICB_STATUS_0, HOST_IPC_FIFO_INT, status))
ivpu_ipc_irq_handler(vdev);
if (REG_TEST_FLD(VPU_40XX_HOST_SS_ICB_STATUS_0, MMU_IRQ_1_INT, status))
ivpu_dbg(vdev, IRQ, "MMU sync complete\n");
if (REG_TEST_FLD(VPU_40XX_HOST_SS_ICB_STATUS_0, MMU_IRQ_2_INT, status))
ivpu_mmu_irq_gerr_handler(vdev);
if (REG_TEST_FLD(VPU_40XX_HOST_SS_ICB_STATUS_0, CPU_INT_REDIRECT_0_INT, status))
irq_wdt_mss_handler(vdev);
if (REG_TEST_FLD(VPU_40XX_HOST_SS_ICB_STATUS_0, CPU_INT_REDIRECT_1_INT, status))
irq_wdt_nce_handler(vdev);
if (REG_TEST_FLD(VPU_40XX_HOST_SS_ICB_STATUS_0, NOC_FIREWALL_INT, status))
irq_noc_firewall_handler(vdev);
return true;
}
static void db_set_37xx(struct ivpu_device *vdev, u32 db_id)
{
u32 reg_stride = VPU_37XX_CPU_SS_DOORBELL_1 - VPU_37XX_CPU_SS_DOORBELL_0;
@@ -1211,10 +950,7 @@ void ivpu_hw_ip_db_set(struct ivpu_device *vdev, u32 db_id)
u32 ivpu_hw_ip_ipc_rx_addr_get(struct ivpu_device *vdev)
{
if (ivpu_hw_ip_gen(vdev) == IVPU_HW_IP_37XX)
return REGV_RD32(VPU_37XX_HOST_SS_TIM_IPC_FIFO_ATM);
else
return REGV_RD32(VPU_40XX_HOST_SS_TIM_IPC_FIFO_ATM);
return REGV_RD32(VPU_37XX_HOST_SS_TIM_IPC_FIFO_ATM);
}
void ivpu_hw_ip_ipc_tx_set(struct ivpu_device *vdev, u32 vpu_addr)
+1 -2
View File
@@ -22,8 +22,7 @@ void ivpu_hw_ip_wdt_disable(struct ivpu_device *vdev);
void ivpu_hw_ip_diagnose_failure(struct ivpu_device *vdev);
u32 ivpu_hw_ip_ipc_rx_count_get(struct ivpu_device *vdev);
void ivpu_hw_ip_irq_clear(struct ivpu_device *vdev);
bool ivpu_hw_ip_irq_handler_37xx(struct ivpu_device *vdev, int irq);
bool ivpu_hw_ip_irq_handler_40xx(struct ivpu_device *vdev, int irq);
bool ivpu_hw_ip_irq_handler(struct ivpu_device *vdev, int irq);
void ivpu_hw_ip_db_set(struct ivpu_device *vdev, u32 db_id);
u32 ivpu_hw_ip_ipc_rx_addr_get(struct ivpu_device *vdev);
void ivpu_hw_ip_ipc_tx_set(struct ivpu_device *vdev, u32 vpu_addr);
@@ -1,18 +1,28 @@
/* SPDX-License-Identifier: GPL-2.0-only */
/*
* Copyright (C) 2020-2023 Intel Corporation
* Copyright (C) 2020-2026 Intel Corporation
*/
#ifndef __IVPU_HW_37XX_REG_H__
#define __IVPU_HW_37XX_REG_H__
#ifndef __IVPU_HW_IP_REG_H__
#define __IVPU_HW_IP_REG_H__
#include <linux/bits.h>
#define VPU_40XX_HOST_SS_CPR_CLK_EN 0x00000080u
#define VPU_40XX_HOST_SS_CPR_CLK_EN_TOP_NOC_MASK BIT_MASK(1)
#define VPU_40XX_HOST_SS_CPR_CLK_EN_DSS_MAS_MASK BIT_MASK(10)
#define VPU_40XX_HOST_SS_CPR_CLK_EN_CSS_MAS_MASK BIT_MASK(11)
#define VPU_37XX_HOST_SS_CPR_CLK_SET 0x00000084u
#define VPU_37XX_HOST_SS_CPR_CLK_SET_TOP_NOC_MASK BIT_MASK(1)
#define VPU_37XX_HOST_SS_CPR_CLK_SET_DSS_MAS_MASK BIT_MASK(10)
#define VPU_37XX_HOST_SS_CPR_CLK_SET_MSS_MAS_MASK BIT_MASK(11)
#define VPU_40XX_HOST_SS_CPR_RST_EN 0x00000090u
#define VPU_40XX_HOST_SS_CPR_RST_EN_TOP_NOC_MASK BIT_MASK(1)
#define VPU_40XX_HOST_SS_CPR_RST_EN_DSS_MAS_MASK BIT_MASK(10)
#define VPU_40XX_HOST_SS_CPR_RST_EN_CSS_MAS_MASK BIT_MASK(11)
#define VPU_37XX_HOST_SS_CPR_RST_SET 0x00000094u
#define VPU_37XX_HOST_SS_CPR_RST_SET_TOP_NOC_MASK BIT_MASK(1)
#define VPU_37XX_HOST_SS_CPR_RST_SET_DSS_MAS_MASK BIT_MASK(10)
@@ -29,6 +39,8 @@
#define VPU_37XX_HOST_SS_HW_VERSION_SOC_NUMBER_MASK GENMASK(15, 8)
#define VPU_37XX_HOST_SS_HW_VERSION_VPU_GENERATION_MASK GENMASK(23, 16)
#define VPU_40XX_HOST_SS_SW_VERSION 0x0000010cu
#define VPU_37XX_HOST_SS_GEN_CTRL 0x00000118u
#define VPU_37XX_HOST_SS_GEN_CTRL_PS_MASK GENMASK(31, 29)
@@ -44,14 +56,17 @@
#define VPU_37XX_TOP_NOC_QREQN 0x00000160u
#define VPU_37XX_TOP_NOC_QREQN_CPU_CTRL_MASK BIT_MASK(0)
#define VPU_37XX_TOP_NOC_QREQN_HOSTIF_L2CACHE_MASK BIT_MASK(1)
#define VPU_40XX_TOP_NOC_QREQN_HOSTIF_L2CACHE_MASK BIT_MASK(2)
#define VPU_37XX_TOP_NOC_QACCEPTN 0x00000164u
#define VPU_37XX_TOP_NOC_QACCEPTN_CPU_CTRL_MASK BIT_MASK(0)
#define VPU_37XX_TOP_NOC_QACCEPTN_HOSTIF_L2CACHE_MASK BIT_MASK(1)
#define VPU_40XX_TOP_NOC_QACCEPTN_HOSTIF_L2CACHE_MASK BIT_MASK(2)
#define VPU_37XX_TOP_NOC_QDENY 0x00000168u
#define VPU_37XX_TOP_NOC_QDENY_CPU_CTRL_MASK BIT_MASK(0)
#define VPU_37XX_TOP_NOC_QDENY_HOSTIF_L2CACHE_MASK BIT_MASK(1)
#define VPU_40XX_TOP_NOC_QDENY_HOSTIF_L2CACHE_MASK BIT_MASK(2)
#define VPU_37XX_HOST_SS_FW_SOC_IRQ_EN 0x00000170u
#define VPU_37XX_HOST_SS_FW_SOC_IRQ_EN_CSS_ROM_CMX_MASK BIT_MASK(0)
@@ -83,6 +98,7 @@
#define VPU_37XX_HOST_SS_ICB_CLEAR_0 0x00010220u
#define VPU_37XX_HOST_SS_ICB_CLEAR_1 0x00010224u
#define VPU_37XX_HOST_SS_ICB_ENABLE_0 0x00010240u
#define VPU_40XX_HOST_SS_ICB_ENABLE_1 0x00010244u
#define VPU_37XX_HOST_SS_TIM_IPC_FIFO_ATM 0x000200f4u
@@ -104,12 +120,29 @@
#define VPU_37XX_HOST_SS_AON_PWR_ISLAND_STATUS0 0x0003002cu
#define VPU_37XX_HOST_SS_AON_PWR_ISLAND_STATUS0_MSS_CPU_MASK BIT_MASK(3)
#define VPU_50XX_HOST_SS_AON_PWR_ISLAND_EN_POST_DLY 0x00030068u
#define VPU_50XX_HOST_SS_AON_PWR_ISLAND_EN_POST_DLY_POST_DLY_MASK GENMASK(7, 0)
#define VPU_50XX_HOST_SS_AON_PWR_ISLAND_EN_POST_DLY_POST1_DLY_MASK GENMASK(15, 8)
#define VPU_50XX_HOST_SS_AON_PWR_ISLAND_EN_POST_DLY_POST2_DLY_MASK GENMASK(23, 16)
#define VPU_50XX_HOST_SS_AON_PWR_ISLAND_STATUS_DLY 0x0003006cu
#define VPU_50XX_HOST_SS_AON_PWR_ISLAND_STATUS_DLY_STATUS_DLY_MASK GENMASK(7, 0)
#define VPU_37XX_HOST_SS_AON_VPU_IDLE_GEN 0x00030200u
#define VPU_37XX_HOST_SS_AON_VPU_IDLE_GEN_EN_MASK BIT_MASK(0)
#define VPU_40XX_HOST_SS_AON_VPU_IDLE_GEN_HW_PG_EN_MASK BIT_MASK(1)
#define VPU_37XX_HOST_SS_AON_DPU_ACTIVE 0x00030204u
#define VPU_37XX_HOST_SS_AON_DPU_ACTIVE_DPU_ACTIVE_MASK BIT_MASK(0)
#define VPU_50XX_HOST_SS_AON_FABRIC_REQ_OVERRIDE 0x00030210u
#define VPU_50XX_HOST_SS_AON_FABRIC_REQ_OVERRIDE_REQ_OVERRIDE_MASK BIT_MASK(0)
#define VPU_40XX_HOST_SS_VERIFICATION_ADDRESS_LO 0x00040040u
#define VPU_40XX_HOST_SS_VERIFICATION_ADDRESS_LO_DONE_MASK BIT_MASK(0)
#define VPU_40XX_HOST_SS_VERIFICATION_ADDRESS_LO_IOSF_RS_ID_MASK GENMASK(2, 1)
#define VPU_40XX_HOST_SS_VERIFICATION_ADDRESS_LO_IMAGE_LOCATION_MASK GENMASK(31, 3)
#define VPU_37XX_HOST_SS_LOADING_ADDRESS_LO 0x00041040u
#define VPU_37XX_HOST_SS_LOADING_ADDRESS_LO_DONE_MASK BIT_MASK(0)
#define VPU_37XX_HOST_SS_LOADING_ADDRESS_LO_IOSF_RS_ID_MASK GENMASK(2, 1)
@@ -126,6 +159,9 @@
#define VPU_37XX_HOST_IF_TCU_PTW_OVERRIDES_NOSNOOP_OVERRIDE_EN_MASK BIT_MASK(3)
#define VPU_37XX_HOST_IF_TCU_PTW_OVERRIDES_AW_NOSNOOP_OVERRIDE_MASK BIT_MASK(4)
#define VPU_37XX_HOST_IF_TCU_PTW_OVERRIDES_AR_NOSNOOP_OVERRIDE_MASK BIT_MASK(5)
#define VPU_40XX_HOST_IF_TCU_PTW_OVERRIDES_SNOOP_OVERRIDE_EN_MASK BIT_MASK(3)
#define VPU_40XX_HOST_IF_TCU_PTW_OVERRIDES_AW_SNOOP_OVERRIDE_MASK BIT_MASK(4)
#define VPU_40XX_HOST_IF_TCU_PTW_OVERRIDES_AR_SNOOP_OVERRIDE_MASK BIT_MASK(5)
#define VPU_37XX_HOST_IF_TCU_PTW_OVERRIDES_PTW_AW_CONTEXT_FLAG_MASK GENMASK(10, 6)
#define VPU_37XX_HOST_IF_TCU_PTW_OVERRIDES_PTW_AR_CONTEXT_FLAG_MASK GENMASK(15, 11)
@@ -141,6 +177,17 @@
#define VPU_37XX_HOST_IF_TBU_MMUSSIDV_TBU4_AWMMUSSIDV_MASK BIT_MASK(8)
#define VPU_37XX_HOST_IF_TBU_MMUSSIDV_TBU4_ARMMUSSIDV_MASK BIT_MASK(9)
#define VPU_40XX_CPU_SS_CPR_NOC_QREQN 0x01010030u
#define VPU_40XX_CPU_SS_CPR_NOC_QREQN_TOP_MMIO_MASK BIT_MASK(0)
#define VPU_40XX_CPU_SS_CPR_NOC_QACCEPTN 0x01010034u
#define VPU_40XX_CPU_SS_CPR_NOC_QACCEPTN_TOP_MMIO_MASK BIT_MASK(0)
#define VPU_40XX_CPU_SS_CPR_NOC_QDENY 0x01010038u
#define VPU_40XX_CPU_SS_CPR_NOC_QDENY_TOP_MMIO_MASK BIT_MASK(0)
#define VPU_40XX_CPU_SS_TIM_PERF_EXT_FREE_CNT 0x01029008u
#define VPU_37XX_CPU_SS_DSU_LEON_RT_BASE 0x04000000u
#define VPU_37XX_CPU_SS_DSU_LEON_RT_DSU_CTRL 0x04000000u
#define VPU_37XX_CPU_SS_DSU_LEON_RT_PC_REG 0x04400010u
@@ -161,18 +208,32 @@
#define VPU_37XX_CPU_SS_MSSCPU_CPR_LEON_RT_VEC_IRQI_RSTVEC_MASK GENMASK(31, 4)
#define VPU_37XX_CPU_SS_TIM_WATCHDOG 0x0602009cu
#define VPU_40XX_CPU_SS_TIM_WATCHDOG 0x0102009cu
#define VPU_37XX_CPU_SS_TIM_WDOG_EN 0x060200a4u
#define VPU_40XX_CPU_SS_TIM_WDOG_EN 0x010200a4u
#define VPU_37XX_CPU_SS_TIM_SAFE 0x060200a8u
#define VPU_40XX_CPU_SS_TIM_SAFE 0x010200a8u
#define VPU_37XX_CPU_SS_TIM_IPC_FIFO 0x060200f0u
#define VPU_40XX_CPU_SS_TIM_IPC_FIFO 0x010200f0u
#define VPU_37XX_CPU_SS_TIM_GEN_CONFIG 0x06021008u
#define VPU_37XX_CPU_SS_TIM_GEN_CONFIG_WDOG_TO_INT_CLR_MASK BIT_MASK(9)
#define VPU_40XX_CPU_SS_TIM_GEN_CONFIG 0x01021008u
#define VPU_40XX_CPU_SS_TIM_GEN_CONFIG_WDOG_TO_INT_CLR_MASK BIT_MASK(9)
#define VPU_37XX_CPU_SS_TIM_PERF_FREE_CNT 0x06029000u
#define VPU_37XX_CPU_SS_DOORBELL_0 0x06300000u
#define VPU_37XX_CPU_SS_DOORBELL_0_SET_MASK BIT_MASK(0)
#define VPU_37XX_CPU_SS_DOORBELL_1 0x06301000u
#define VPU_40XX_CPU_SS_DOORBELL_0 0x01300000u
#define VPU_40XX_CPU_SS_DOORBELL_0_SET_MASK BIT_MASK(0)
#endif /* __IVPU_HW_37XX_REG_H__ */
#define VPU_37XX_CPU_SS_DOORBELL_1 0x06301000u
#define VPU_40XX_CPU_SS_DOORBELL_1 0x01301000u
#endif /* __IVPU_HW_IP_REG_H__ */
+5 -6
View File
@@ -509,7 +509,7 @@ int ivpu_ipc_init(struct ivpu_device *vdev)
if (!ipc->mem_rx) {
ivpu_err(vdev, "Failed to allocate mem_rx\n");
ret = -ENOMEM;
goto err_free_tx;
goto err_free_bo;
}
ipc->mm_tx = devm_gen_pool_create(vdev->drm.dev, __ffs(IVPU_IPC_ALIGNMENT),
@@ -517,13 +517,13 @@ int ivpu_ipc_init(struct ivpu_device *vdev)
if (IS_ERR(ipc->mm_tx)) {
ret = PTR_ERR(ipc->mm_tx);
ivpu_err(vdev, "Failed to create gen pool, %pe\n", ipc->mm_tx);
goto err_free_rx;
goto err_free_bo;
}
ret = gen_pool_add(ipc->mm_tx, ipc->mem_tx->vpu_addr, ivpu_bo_size(ipc->mem_tx), -1);
if (ret) {
ivpu_err(vdev, "gen_pool_add failed, ret %d\n", ret);
goto err_free_rx;
goto err_free_bo;
}
spin_lock_init(&ipc->cons_lock);
@@ -532,14 +532,13 @@ int ivpu_ipc_init(struct ivpu_device *vdev)
ret = drmm_mutex_init(&vdev->drm, &ipc->lock);
if (ret) {
ivpu_err(vdev, "Failed to initialize ipc->lock, ret %d\n", ret);
goto err_free_rx;
goto err_free_bo;
}
ivpu_ipc_reset(vdev);
return 0;
err_free_rx:
err_free_bo:
ivpu_bo_free(ipc->mem_rx);
err_free_tx:
ivpu_bo_free(ipc->mem_tx);
err_destroy_cache:
kmem_cache_destroy(ipc->rx_msg_cache);
+93 -5
View File
@@ -64,10 +64,8 @@ err_free_primary:
static void ivpu_preemption_buffers_free(struct ivpu_device *vdev,
struct ivpu_file_priv *file_priv, struct ivpu_cmdq *cmdq)
{
if (cmdq->primary_preempt_buf)
ivpu_bo_free(cmdq->primary_preempt_buf);
if (cmdq->secondary_preempt_buf)
ivpu_bo_free(cmdq->secondary_preempt_buf);
ivpu_bo_free(cmdq->primary_preempt_buf);
ivpu_bo_free(cmdq->secondary_preempt_buf);
}
static int ivpu_preemption_job_init(struct ivpu_device *vdev, struct ivpu_file_priv *file_priv,
@@ -323,6 +321,11 @@ static inline u8 ivpu_job_to_jsm_priority(u8 priority)
return priority - 1;
}
static inline u8 ivpu_jsm_to_job_priority(u8 priority)
{
return priority + 1;
}
static void ivpu_cmdq_destroy(struct ivpu_file_priv *file_priv, struct ivpu_cmdq *cmdq)
{
lockdep_assert_held(&file_priv->lock);
@@ -1126,6 +1129,86 @@ int ivpu_cmdq_destroy_ioctl(struct drm_device *dev, void *data, struct drm_file
return ret;
}
static int ivpu_cmdq_priority_args_check(struct ivpu_device *vdev, u32 cmdq_id, u64 priority)
{
if (!ivpu_is_capable(vdev, DRM_IVPU_CAP_CMDQ_SET_PRIORITY)) {
ivpu_dbg(vdev, IOCTL, "Command queue priority not supported\n");
return -ENODEV;
}
if (cmdq_id < IVPU_CMDQ_MIN_ID || cmdq_id > IVPU_CMDQ_MAX_ID) {
ivpu_dbg(vdev, IOCTL, "Invalid command queue ID %u\n", cmdq_id);
return -EINVAL;
}
if (priority > DRM_IVPU_JOB_PRIORITY_REALTIME)
return -EINVAL;
return 0;
}
int ivpu_cmdq_get_priority(struct ivpu_file_priv *file_priv, u32 cmdq_id, u64 *priority)
{
struct ivpu_device *vdev = file_priv->vdev;
struct ivpu_cmdq *cmdq;
int ret;
ret = ivpu_cmdq_priority_args_check(vdev, cmdq_id, DRM_IVPU_JOB_PRIORITY_NORMAL);
if (ret)
return ret;
guard(mutex)(&file_priv->lock);
cmdq = ivpu_cmdq_acquire(file_priv, cmdq_id);
if (!cmdq || cmdq->is_legacy)
return -EINVAL;
*priority = ivpu_jsm_to_job_priority(cmdq->priority);
return 0;
}
int ivpu_cmdq_set_priority(struct ivpu_file_priv *file_priv, u32 cmdq_id, u64 priority)
{
struct ivpu_device *vdev = file_priv->vdev;
struct ivpu_cmdq *cmdq;
u8 jsm_priority;
int ret = 0;
ret = ivpu_cmdq_priority_args_check(vdev, cmdq_id, priority);
if (ret)
return ret;
ret = ivpu_rpm_get(vdev);
if (ret < 0)
return ret;
mutex_lock(&file_priv->lock);
cmdq = ivpu_cmdq_acquire(file_priv, cmdq_id);
if (!cmdq || cmdq->is_legacy) {
ret = -EINVAL;
goto out;
}
jsm_priority = ivpu_job_to_jsm_priority(priority);
if (cmdq->priority == jsm_priority)
goto out;
if (cmdq->db_id) {
ret = ivpu_jsm_hws_set_context_sched_properties(vdev, file_priv->ctx.id, cmdq_id,
jsm_priority);
if (ret)
goto out;
}
cmdq->priority = jsm_priority;
out:
mutex_unlock(&file_priv->lock);
ivpu_rpm_put(vdev);
return ret;
}
static void
ivpu_job_done_callback(struct ivpu_device *vdev, struct ivpu_ipc_hdr *ipc_hdr,
struct vpu_jsm_msg *jsm_msg)
@@ -1215,9 +1298,12 @@ void ivpu_context_abort_work_fn(struct work_struct *work)
unsigned long ctx_id;
unsigned long id;
if (drm_WARN_ON(&vdev->drm, pm_runtime_get_if_active(vdev->drm.dev) <= 0))
if (!down_read_trylock(&vdev->pm->reset_lock))
return;
if (drm_WARN_ON(&vdev->drm, pm_runtime_get_if_active(vdev->drm.dev) <= 0))
goto unlock;
if (vdev->fw->sched_mode == VPU_SCHEDULING_MODE_HW)
if (reset_engine_and_mark_faulty_contexts(vdev))
goto runtime_put;
@@ -1258,4 +1344,6 @@ void ivpu_context_abort_work_fn(struct work_struct *work)
runtime_put:
pm_runtime_put_autosuspend(vdev->drm.dev);
unlock:
up_read(&vdev->pm->reset_lock);
}
+4 -1
View File
@@ -1,6 +1,6 @@
/* SPDX-License-Identifier: GPL-2.0-only */
/*
* Copyright (C) 2020-2025 Intel Corporation
* Copyright (C) 2020-2026 Intel Corporation
*/
#ifndef __IVPU_JOB_H__
@@ -81,6 +81,9 @@ int ivpu_cmdq_create_ioctl(struct drm_device *dev, void *data, struct drm_file *
int ivpu_cmdq_destroy_ioctl(struct drm_device *dev, void *data, struct drm_file *file);
int ivpu_cmdq_submit_ioctl(struct drm_device *dev, void *data, struct drm_file *file);
int ivpu_cmdq_get_priority(struct ivpu_file_priv *file_priv, u32 cmdq_id, u64 *priority);
int ivpu_cmdq_set_priority(struct ivpu_file_priv *file_priv, u32 cmdq_id, u64 priority);
void ivpu_context_abort_locked(struct ivpu_file_priv *file_priv);
void ivpu_cmdq_release_all_locked(struct ivpu_file_priv *file_priv);
+4 -8
View File
@@ -220,9 +220,6 @@
#define IVPU_MMU_STE_1_S1DSS GENMASK_ULL(1, 0)
#define IVPU_MMU_STE_1_S1DSS_TERMINATE 0x0
#define IVPU_MMU_REG_TIMEOUT_US (10 * USEC_PER_MSEC)
#define IVPU_MMU_QUEUE_TIMEOUT_US (100 * USEC_PER_MSEC)
#define IVPU_MMU_GERROR_ERR_MASK ((REG_FLD(IVPU_MMU_REG_GERROR, CMDQ)) | \
(REG_FLD(IVPU_MMU_REG_GERROR, EVTQ_ABT)) | \
(REG_FLD(IVPU_MMU_REG_GERROR, PRIQ_ABT)) | \
@@ -438,14 +435,14 @@ static int ivpu_mmu_reg_write_cr0(struct ivpu_device *vdev, u32 val)
{
REGV_WR32(IVPU_MMU_REG_CR0, val);
return REGV_POLL_FLD(IVPU_MMU_REG_CR0ACK, VAL, val, IVPU_MMU_REG_TIMEOUT_US);
return REGV_POLL_FLD(IVPU_MMU_REG_CR0ACK, VAL, val, vdev->timeout.mmu_reg);
}
static int ivpu_mmu_reg_write_irq_ctrl(struct ivpu_device *vdev, u32 val)
{
REGV_WR32(IVPU_MMU_REG_IRQ_CTRL, val);
return REGV_POLL_FLD(IVPU_MMU_REG_IRQ_CTRLACK, VAL, val, IVPU_MMU_REG_TIMEOUT_US);
return REGV_POLL_FLD(IVPU_MMU_REG_IRQ_CTRLACK, VAL, val, vdev->timeout.mmu_reg);
}
static int ivpu_mmu_irqs_setup(struct ivpu_device *vdev)
@@ -465,8 +462,7 @@ static int ivpu_mmu_cmdq_wait_for_cons(struct ivpu_device *vdev)
struct ivpu_mmu_queue *cmdq = &vdev->mmu->cmdq;
int ret;
ret = REGV_POLL_FLD(IVPU_MMU_REG_CMDQ_CONS, VAL, cmdq->prod,
IVPU_MMU_QUEUE_TIMEOUT_US);
ret = REGV_POLL_FLD(IVPU_MMU_REG_CMDQ_CONS, VAL, cmdq->prod, vdev->timeout.mmu_queue);
if (ret)
return ret;
@@ -880,7 +876,7 @@ static int ivpu_mmu_evtq_set(struct ivpu_device *vdev, bool enable)
val = REG_CLR_FLD(IVPU_MMU_REG_CR0, EVTQEN, val);
REGV_WR32(IVPU_MMU_REG_CR0, val);
return REGV_POLL_FLD(IVPU_MMU_REG_CR0ACK, VAL, val, IVPU_MMU_REG_TIMEOUT_US);
return REGV_POLL_FLD(IVPU_MMU_REG_CR0ACK, VAL, val, vdev->timeout.mmu_reg);
}
static int ivpu_mmu_evtq_enable(struct ivpu_device *vdev)

Some files were not shown because too many files have changed in this diff Show More