Commit Graph

707 Commits

Author SHA1 Message Date
Huacai Chen
d55fe7b7cf PCI: Add ACS quirk for Loongson PCIe
commit 1f3303aa92 upstream.

Loongson PCIe Root Ports don't advertise an ACS capability, but they do not
allow peer-to-peer transactions between Root Ports. Add an ACS quirk so
each Root Port can be in a separate IOMMU group.

Signed-off-by: Xianglai Li <lixianglai@loongson.cn>
Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Cc: stable@vger.kernel.org
Link: https://patch.msgid.link/20250403040756.720409-1-chenhuacai@loongson.cn
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2025-06-27 11:07:31 +01:00
Rakesh Babu Saladi
e86cc66b10 PCI: switchtec: Add Microchip PCI100X device IDs
[ Upstream commit a3282f84b2 ]

Add Microchip parts to the Device ID table so the driver supports PCI100x
devices.

Add a new macro to quirk the Microchip Switchtec PCI100x parts to allow DMA
access via NTB to work when the IOMMU is turned on.

PCI100x family has 6 variants; each variant is designed for different
application usages, different port counts and lane counts:

  PCI1001 has 1 x4 upstream port and 3 x4 downstream ports
  PCI1002 has 1 x4 upstream port and 4 x2 downstream ports
  PCI1003 has 2 x4 upstream ports, 2 x2 upstream ports, and 2 x2
    downstream ports
  PCI1004 has 4 x4 upstream ports
  PCI1005 has 1 x4 upstream port and 6 x2 downstream ports
  PCI1006 has 6 x2 upstream ports and 2 x2 downstream ports

[Historical note: these parts use PCI_VENDOR_ID_EFAR (0x1055), from EFAR
Microsystems, which was acquired in 1996 by Standard Microsystems Corp,
which was acquired by Microchip Technology in 2012.  The PCI-SIG confirms
that Vendor ID 0x1055 is assigned to Microchip even though it's not
visible via https://pcisig.com/membership/member-companies]

Link: https://lore.kernel.org/r/20250120095524.243103-1-Saladi.Rakeshbabu@microchip.com
Signed-off-by: Rakesh Babu Saladi <Saladi.Rakeshbabu@microchip.com>
[bhelgaas: Vendor ID history]
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Acked-By: Logan Gunthorpe <logang@deltatee.com>
Signed-off-by: Sasha Levin <sashal@kernel.org>
2025-02-21 13:49:59 +01:00
Takashi Iwai
d2eb7e8732 PCI/DPC: Quirk PIO log size for Intel Raptor Lake-P
[ Upstream commit b198499c7d ]

Apparently the Raptor Lake-P reference firmware configures the PIO log size
correctly, but some vendor BIOSes, including at least ASUSTeK COMPUTER INC.
Zenbook UX3402VA_UX3402VA, do not.

Apply the quirk for Raptor Lake-P.  This prevents kernel complaints like:

  DPC: RP PIO log size 0 is invalid

and also enables the DPC driver to dump the RP PIO Log registers when DPC
is triggered.

Note that the bug report also mentions 8086:a76e, which has been already
added by 627c6db207 ("PCI/DPC: Quirk PIO log size for Intel Raptor Lake
Root Ports").

Link: https://lore.kernel.org/r/20250102164315.7562-1-tiwai@suse.de
Link: https://bugzilla.suse.com/show_bug.cgi?id=1234623
Signed-off-by: Takashi Iwai <tiwai@suse.de>
[bhelgaas: commit log]
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Signed-off-by: Krzysztof Wilczyński <kwilczynski@kernel.org>
Signed-off-by: Sasha Levin <sashal@kernel.org>
2025-02-21 13:49:59 +01:00
Ajit Khaparde
f6720b1362 PCI: Add ACS quirk for Broadcom BCM5760X NIC
[ Upstream commit 524e057b2d ]

The Broadcom BCM5760X NIC may be a multi-function device.

While it does not advertise an ACS capability, peer-to-peer transactions
are not possible between the individual functions. So it is ok to treat
them as fully isolated.

Add an ACS quirk for this device so the functions can be in independent
IOMMU groups and attached individually to userspace applications using
VFIO.

[kwilczynski: commit log]
Link: https://lore.kernel.org/linux-pci/20240510204228.73435-1-ajit.khaparde@broadcom.com
Signed-off-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
Signed-off-by: Krzysztof Wilczyński <kwilczynski@kernel.org>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Andy Gospodarek <gospo@broadcom.com>
Signed-off-by: Sasha Levin <sashal@kernel.org>
2024-12-27 13:52:52 +01:00
Mengyuan Lou
d51345f75f PCI: Add ACS quirk for Wangxun FF5xxx NICs
[ Upstream commit aa46a3736a ]

Wangxun FF5xxx NICs are similar to SFxxx, RP1000 and RP2000 NICs.  They may
be multi-function devices, but they do not advertise an ACS capability.

But the hardware does isolate FF5xxx functions as though it had an ACS
capability and PCI_ACS_RR and PCI_ACS_CR were set in the ACS Control
register, i.e., all peer-to-peer traffic is directed upstream instead of
being routed internally.

Add ACS quirk for FF5xxx NICs in pci_quirk_wangxun_nic_acs() so the
functions can be in independent IOMMU groups.

Link: https://lore.kernel.org/r/E16053DB2B80E9A5+20241115024604.30493-1-mengyuanlou@net-swift.com
Signed-off-by: Mengyuan Lou <mengyuanlou@net-swift.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Signed-off-by: Sasha Levin <sashal@kernel.org>
2024-12-14 19:54:48 +01:00
Alex Williamson
a4babb895c PCI: Mark Creative Labs EMU20k2 INTx masking as broken
[ Upstream commit 2910306655 ]

Per user reports, the Creative Labs EMU20k2 (Sound Blaster X-Fi
Titanium Series) generates spurious interrupts when used with
vfio-pci unless DisINTx masking support is disabled.

Thus, quirk the device to mark INTx masking as broken.

Closes: https://lore.kernel.org/all/VI1PR10MB8207C507DB5420AB4C7281E0DB9A2@VI1PR10MB8207.EURPRD10.PROD.OUTLOOK.COM
Link: https://lore.kernel.org/linux-pci/20240912215331.839220-1-alex.williamson@redhat.com
Reported-by: zdravko delineshev <delineshev@outlook.com>
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
[kwilczynski: commit log]
Signed-off-by: Krzysztof Wilczyński <kwilczynski@kernel.org>
Signed-off-by: Sasha Levin <sashal@kernel.org>
2024-10-17 15:22:14 +02:00
Subramanian Ananthanarayanan
9af86455eb PCI: Add ACS quirk for Qualcomm SA8775P
[ Upstream commit 026f84d3fa ]

The Qualcomm SA8775P root ports don't advertise an ACS capability, but they
do provide ACS-like features to disable peer transactions and validate bus
numbers in requests.

Thus, add an ACS quirk for the SA8775P.

Link: https://lore.kernel.org/linux-pci/20240906052228.1829485-1-quic_skananth@quicinc.com
Signed-off-by: Subramanian Ananthanarayanan <quic_skananth@quicinc.com>
Signed-off-by: Krzysztof Wilczyński <kwilczynski@kernel.org>
Signed-off-by: Sasha Levin <sashal@kernel.org>
2024-10-17 15:22:14 +02:00
WangYuli
90be6ffc6a PCI: Add function 0 DMA alias quirk for Glenfly Arise chip
[ Upstream commit 9246b487ab ]

Add DMA support for audio function of Glenfly Arise chip, which uses
Requester ID of function 0.

Link: https://lore.kernel.org/r/CA2BBD087345B6D1+20240823095708.3237375-1-wangyuli@uniontech.com
Signed-off-by: SiyuLi <siyuli@glenfly.com>
Signed-off-by: WangYuli <wangyuli@uniontech.com>
[bhelgaas: lower-case hex to match local code, drop unused Device IDs]
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Takashi Iwai <tiwai@suse.de>
Signed-off-by: Sasha Levin <sashal@kernel.org>
2024-10-17 15:22:14 +02:00
Ilpo Järvinen
39f932d295 PCI: Simplify pcie_capability_clear_and_set_word() to ..._clear_word()
[ Upstream commit 0fce6e5c87 ]

When using pcie_capability_clear_and_set_word() but not actually *setting*
anything, use pcie_capability_clear_word() instead.

Link: https://lore.kernel.org/r/20231026121924.2164-1-ilpo.jarvinen@linux.intel.com
Link: https://lore.kernel.org/r/20231026121924.2164-2-ilpo.jarvinen@linux.intel.com
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
[bhelgaas: squash]
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Signed-off-by: Sasha Levin <sashal@kernel.org>
2024-04-27 17:07:11 +02:00
Bjorn Helgaas
636f0fdb36 PCI/DPC: Use FIELD_GET()
[ Upstream commit 9a9eec4765 ]

Use FIELD_GET() to remove dependencies on the field position, i.e., the
shift value. No functional change intended.

Link: https://lore.kernel.org/r/20231018113254.17616-5-ilpo.jarvinen@linux.intel.com
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Signed-off-by: Sasha Levin <sashal@kernel.org>
2024-04-27 17:07:11 +02:00
Kelvin Cao
bbdfa14482 PCI: switchtec: Add support for PCIe Gen5 devices
[ Upstream commit 0fb53e6470 ]

Advertise support of Gen5 devices in the driver's device ID table and
add the same IDs for the switchtec quirks. Also update driver code to
accommodate them.

Link: https://lore.kernel.org/r/20230624000003.2315364-3-kelvin.cao@microchip.com
Signed-off-by: Kelvin Cao <kelvin.cao@microchip.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Logan Gunthorpe <logang@deltatee.com>
Signed-off-by: Sasha Levin <sashal@kernel.org>
2024-04-27 17:07:10 +02:00
Maciej W. Rozycki
89a9196aec PCI: Execute quirk_enable_clear_retrain_link() earlier
[ Upstream commit 07a8d698de ]

Make quirk_enable_clear_retrain_link() an early quirk so that any later
fixups can rely on dev->clear_retrain_link to have been already
initialised.

[bhelgaas: reorder to just before it becomes possible to call
pcie_retrain_link() earlier]
Link: https://lore.kernel.org/r/alpine.DEB.2.21.2305310049000.59226@angie.orcam.me.uk
Signed-off-by: Maciej W. Rozycki <macro@orcam.me.uk>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Signed-off-by: Sasha Levin <sashal@kernel.org>
2024-04-27 17:07:10 +02:00
Mike Pastore
f4aae2afe2 PCI: Delay after FLR of Solidigm P44 Pro NVMe
[ Upstream commit 0ac448e0d2 ]

Prevent KVM hang when a Solidgm P44 Pro NVMe is passed through to a guest
via IOMMU and the guest is subsequently rebooted.

A similar issue was identified and patched by 51ba09452d ("PCI: Delay
after FLR of Intel DC P3700 NVMe") and the same fix can be applied for this
case. (Intel spun off their NAND and SSD business as Solidigm and sold it
to SK Hynix in late 2021.)

Link: https://lore.kernel.org/r/20230507073519.9737-1-mike@oobak.org
Signed-off-by: Mike Pastore <mike@oobak.org>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Signed-off-by: Sasha Levin <sashal@kernel.org>
2024-04-27 17:07:10 +02:00
Alvaro Karsz
57aadcc028 PCI: Avoid FLR for SolidRun SNET DPU rev 1
[ Upstream commit d089d69cc1 ]

This patch fixes a FLR bug on the SNET DPU rev 1 by setting the
PCI_DEV_FLAGS_NO_FLR_RESET flag.

As there is a quirk to avoid FLR (quirk_no_flr), I added a new quirk
to check the rev ID before calling to quirk_no_flr.

Without this patch, a SNET DPU rev 1 may hang when FLR is applied.

Signed-off-by: Alvaro Karsz <alvaro.karsz@solid-run.com>
Acked-by: Bjorn Helgaas <bhelgaas@google.com>
Message-Id: <20230110165638.123745-3-alvaro.karsz@solid-run.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Sasha Levin <sashal@kernel.org>
2024-04-27 17:07:09 +02:00
Paul Menzel
c9ef367b3e PCI/DPC: Quirk PIO log size for Intel Raptor Lake Root Ports
[ Upstream commit 627c6db207 ]

Commit 5459c0b704 ("PCI/DPC: Quirk PIO log size for certain Intel Root
Ports") and commit 3b8803494a ("PCI/DPC: Quirk PIO log size for Intel Ice
Lake Root Ports") add quirks for Ice, Tiger and Alder Lake Root Ports.
System firmware for Raptor Lake still has the bug, so Linux logs the
warning below on several Raptor Lake systems like Dell Precision 3581 with
Intel Raptor Lake processor (0W18NX) system firmware/BIOS version 1.10.1.

  pci 0000:00:07.0: [8086:a76e] type 01 class 0x060400
  pci 0000:00:07.0: DPC: RP PIO log size 0 is invalid
  pci 0000:00:07.1: [8086:a73f] type 01 class 0x060400
  pci 0000:00:07.1: DPC: RP PIO log size 0 is invalid

Apply the quirk for Raptor Lake Root Ports as well.

This also enables the DPC driver to dump the RP PIO Log registers when DPC
is triggered.

Link: https://lore.kernel.org/r/20240305113057.56468-1-pmenzel@molgen.mpg.de
Reported-by: Niels van Aert <nvaert1986@hotmail.com>
Closes: https://bugzilla.kernel.org/show_bug.cgi?id=218560
Signed-off-by: Paul Menzel <pmenzel@molgen.mpg.de>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Cc: <stable@vger.kernel.org>
Cc: Mika Westerberg <mika.westerberg@linux.intel.com>
Cc: Niels van Aert <nvaert1986@hotmail.com>
Signed-off-by: Sasha Levin <sashal@kernel.org>
2024-04-03 15:19:31 +02:00
Jörg Wedekind
ce106d8ef0 PCI: Mark 3ware-9650SE Root Port Extended Tags as broken
[ Upstream commit baf67aefbe ]

Per PCIe r6.1, sec 2.2.6.2 and 7.5.3.4, a Requester may not use 8-bit Tags
unless its Extended Tag Field Enable is set, but all Receivers/Completers
must handle 8-bit Tags correctly regardless of their Extended Tag Field
Enable.

Some devices do not handle 8-bit Tags as Completers, so add a quirk for
them.  If we find such a device, we disable Extended Tags for the entire
hierarchy to make peer-to-peer DMA possible.

The 3ware 9650SE seems to have issues with handling 8-bit tags. Mark it as
broken.

This fixes PCI Parity Errors like :

  3w-9xxx: scsi0: ERROR: (0x06:0x000C): PCI Parity Error: clearing.
  3w-9xxx: scsi0: ERROR: (0x06:0x000D): PCI Abort: clearing.
  3w-9xxx: scsi0: ERROR: (0x06:0x000E): Controller Queue Error: clearing.
  3w-9xxx: scsi0: ERROR: (0x06:0x0010): Microcontroller Error: clearing.

Link: https://lore.kernel.org/r/20240219132811.8351-1-joerg@wedekind.de
Fixes: 60db3a4d8c ("PCI: Enable PCIe Extended Tags if supported")
Closes: https://bugzilla.kernel.org/show_bug.cgi?id=202425
Signed-off-by: Jörg Wedekind <joerg@wedekind.de>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Signed-off-by: Sasha Levin <sashal@kernel.org>
2024-03-26 18:20:50 -04:00
Guilherme G. Piccoli
5e0160dab1 PCI: Only override AMD USB controller if required
[ Upstream commit e585a37e50 ]

By running a Van Gogh device (Steam Deck), the following message
was noticed in the kernel log:

  pci 0000:04:00.3: PCI class overridden (0x0c03fe -> 0x0c03fe) so dwc3 driver can claim this instead of xhci

Effectively this means the quirk executed but changed nothing, since the
class of this device was already the proper one (likely adjusted by newer
firmware versions).

Check and perform the override only if necessary.

Link: https://lore.kernel.org/r/20231120160531.361552-1-gpiccoli@igalia.com
Signed-off-by: Guilherme G. Piccoli <gpiccoli@igalia.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Cc: Huang Rui <ray.huang@amd.com>
Cc: Vicki Pfau <vi@endrift.com>
Signed-off-by: Sasha Levin <sashal@kernel.org>
2024-02-05 20:12:58 +00:00
Ido Schimmel
c0d5a69322 PCI: Add no PM reset quirk for NVIDIA Spectrum devices
[ Upstream commit 3ed48c80b2 ]

Spectrum-{1,2,3,4} devices report that a D3hot->D0 transition causes a
reset (i.e., they advertise NoSoftRst-). However, this transition does
not have any effect on the device: It continues to be operational and
network ports remain up. Advertising this support makes it seem as if a
PM reset is viable for these devices. Mark it as unavailable to skip it
when testing reset methods.

Before:

 # cat /sys/bus/pci/devices/0000\:03\:00.0/reset_method
 pm bus

After:

 # cat /sys/bus/pci/devices/0000\:03\:00.0/reset_method
 bus

Signed-off-by: Ido Schimmel <idosch@nvidia.com>
Acked-by: Bjorn Helgaas <bhelgaas@google.com>
Signed-off-by: Petr Machata <petrm@nvidia.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
Signed-off-by: Sasha Levin <sashal@kernel.org>
2024-02-05 20:12:50 +00:00
LeoLiuoc
d08e756e25 PCI: Add ACS quirk for more Zhaoxin Root Ports
commit e367e3c765 upstream.

Add more Root Port Device IDs to pci_quirk_zhaoxin_pcie_ports_acs() for
some new Zhaoxin platforms.

Fixes: 299bd044a6 ("PCI: Add ACS quirk for Zhaoxin Root/Downstream Ports")
Link: https://lore.kernel.org/r/20231211091543.735903-1-LeoLiu-oc@zhaoxin.com
Signed-off-by: LeoLiuoc <LeoLiu-oc@zhaoxin.com>
[bhelgaas: update subject, drop changelog, add Fixes, add stable tag, fix
whitespace, wrap code comment]
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Cc: <stable@vger.kernel.org>	# 5.7
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2024-01-20 11:50:11 +01:00
Lukas Wunner
1c8f75ee92 PCI: Lengthen reset delay for VideoPropulsion Torrent QN16e card
[ Upstream commit c9260693aa ]

Commit ac91e69805 ("PCI: Unify delay handling for reset and resume")
shortened an unconditional 1 sec delay after a Secondary Bus Reset to 100
msec for PCIe (per PCIe r6.1 sec 6.6.1).  The 1 sec delay is only required
for Conventional PCI.

But it turns out that there are PCIe devices which require a longer delay
than prescribed before first config space access after reset recovery or
resume from D3cold:

Chad reports that a "VideoPropulsion Torrent QN16e" MPEG QAM Modulator
"raises a PCI system error (PERR), as reported by the IPMI event log, and
the hardware itself would suffer a catastrophic event, cycling the server"
unless the longer delay is observed.

The card is specified to conform to PCIe r1.0 and indeed only supports Gen1
speed (2.5 GT/s) according to lspci.  PCIe r1.0 sec 7.6 prescribes the same
100 msec delay as PCIe r6.1 sec 6.6.1:

  To allow components to perform internal initialization, system software
  must wait for at least 100 ms from the end of a reset (cold/warm/hot)
  before it is permitted to issue Configuration Requests

The behavior of the Torrent QN16e card thus appears to be a quirk.  Treat
it as such and lengthen the reset delay for this specific device.

Fixes: ac91e69805 ("PCI: Unify delay handling for reset and resume")
Link: https://lore.kernel.org/r/47727e792c7f0282dc144e3ec8ce8eb6e713394e.1695304512.git.lukas@wunner.de
Reported-by: Chad Schroeder <CSchroeder@sonifi.com>
Closes: https://lore.kernel.org/linux-pci/DM6PR16MB2844903E34CAB910082DF019B1FAA@DM6PR16MB2844.namprd16.prod.outlook.com/
Tested-by: Chad Schroeder <CSchroeder@sonifi.com>
Signed-off-by: Lukas Wunner <lukas@wunner.de>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Cc: stable@vger.kernel.org # v5.4+
Signed-off-by: Sasha Levin <sashal@kernel.org>
2023-12-08 08:51:18 +01:00
Bartosz Pawlowski
6f9557a5c9 PCI: Disable ATS for specific Intel IPU E2000 devices
[ Upstream commit a18615b1cf ]

Due to a hardware issue in A and B steppings of Intel IPU E2000, it expects
wrong endianness in ATS invalidation message body. This problem can lead to
outdated translations being returned as valid and finally cause system
instability.

To prevent such issues, add quirk_intel_e2000_no_ats() to disable ATS for
vulnerable IPU E2000 devices.

Link: https://lore.kernel.org/r/20230908143606.685930-3-bartosz.pawlowski@intel.com
Signed-off-by: Bartosz Pawlowski <bartosz.pawlowski@intel.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Reviewed-by: Alexander Lobakin <aleksander.lobakin@intel.com>
Signed-off-by: Sasha Levin <sashal@kernel.org>
2023-11-28 17:07:00 +00:00
Bartosz Pawlowski
161767bb7c PCI: Extract ATS disabling to a helper function
[ Upstream commit f18b1137d3 ]

Introduce quirk_no_ats() helper function to provide a standard way to
disable ATS capability in PCI quirks.

Suggested-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Link: https://lore.kernel.org/r/20230908143606.685930-2-bartosz.pawlowski@intel.com
Signed-off-by: Bartosz Pawlowski <bartosz.pawlowski@intel.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Signed-off-by: Sasha Levin <sashal@kernel.org>
2023-11-28 17:07:00 +00:00
Ilpo Järvinen
dabc0ee84d PCI: Do error check on own line to split long "if" conditions
[ Upstream commit d15f18053e ]

Placing PCI error code check inside "if" condition usually results in need
to split lines. Combined with additional conditions the "if" condition
becomes messy.

Convert to the usual error handling pattern with an additional variable to
improve code readability. In addition, reverse the logic in
pci_find_vsec_capability() to get rid of &&.

No functional changes intended.

Link: https://lore.kernel.org/r/20230911125354.25501-5-ilpo.jarvinen@linux.intel.com
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
[bhelgaas: PCI_POSSIBLE_ERROR()]
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Signed-off-by: Sasha Levin <sashal@kernel.org>
2023-11-28 17:07:00 +00:00
Vicki Pfau
b25a2f2470 PCI: Prevent xHCI driver from claiming AMD VanGogh USB3 DRD device
commit 7e6f3b6d2c upstream.

The AMD VanGogh SoC contains a DesignWare USB3 Dual-Role Device that can be
operated as either a USB Host or a USB Device, similar to on the AMD Nolan
platform.

be6646bfba ("PCI: Prevent xHCI driver from claiming AMD Nolan USB3 DRD
device") added a quirk to let the dwc3 driver claim the Nolan device since
it provides more specific support.

Extend that quirk to include the VanGogh SoC USB3 device.

Link: https://lore.kernel.org/r/20230927202212.2388216-1-vi@endrift.com
Signed-off-by: Vicki Pfau <vi@endrift.com>
[bhelgaas: include be6646bfba reference, add stable tag]
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Cc: stable@vger.kernel.org	# v3.19+
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2023-11-08 14:11:03 +01:00
Bjorn Helgaas
1654635bed Revert "PCI: Mark NVIDIA T4 GPUs to avoid bus reset"
commit 5260bd6d36 upstream.

This reverts commit d5af729dc2.

d5af729dc2 ("PCI: Mark NVIDIA T4 GPUs to avoid bus reset") avoided
Secondary Bus Reset on the T4 because the reset seemed to not work when the
T4 was directly attached to a Root Port.

But NVIDIA thinks the issue is probably related to some issue with the Root
Port, not with the T4.  The T4 provides neither PM nor FLR reset, so
masking bus reset compromises this device for assignment scenarios.

Revert d5af729dc2 as requested by Wu Zongyong.  This will leave SBR
broken in the specific configuration Wu tested, as it was in v6.5, so Wu
will debug that further.

Link: https://lore.kernel.org/r/ZPqMCDWvITlOLHgJ@wuzongyong-alibaba
Link: https://lore.kernel.org/r/20230908201104.GA305023@bhelgaas
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2023-09-13 09:43:02 +02:00
Wu Zongyong
8562df72cf PCI: Mark NVIDIA T4 GPUs to avoid bus reset
[ Upstream commit d5af729dc2 ]

NVIDIA T4 GPUs do not work with SBR. This problem is found when the T4 card
is direct attached to a Root Port only. Avoid bus reset by marking T4 GPUs
PCI_DEV_FLAGS_NO_BUS_RESET.

Fixes: 4c207e7121 ("PCI: Mark some NVIDIA GPUs to avoid bus reset")
Link: https://lore.kernel.org/r/2dcebea53a6eb9bd212ec6d8974af2e5e0333ef6.1681129861.git.wuzongyong@linux.alibaba.com
Signed-off-by: Wu Zongyong <wuzongyong@linux.alibaba.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Signed-off-by: Sasha Levin <sashal@kernel.org>
2023-09-13 09:42:46 +02:00
Robin Murphy
c459365ec7 PCI: Add function 1 DMA alias quirk for Marvell 88SE9235
commit 88d341716b upstream.

Marvell's own product brief implies the 92xx series are a closely related
family, and sure enough it turns out that 9235 seems to need the same quirk
as the other three, although possibly only when certain ports are used.

Link: https://lore.kernel.org/linux-iommu/2a699a99-545c-1324-e052-7d2f41fed1ae@yahoo.co.uk/
Link: https://lore.kernel.org/r/731507e05d70239aec96fcbfab6e65d8ce00edd2.1686157165.git.robin.murphy@arm.com
Reported-by: Jason Adriaanse <jason_a69@yahoo.co.uk>
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Cc: stable@vger.kernel.org
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2023-07-23 13:49:36 +02:00
Mika Westerberg
cdf9cfc1bb PCI/DPC: Quirk PIO log size for Intel Ice Lake Root Ports
[ Upstream commit 3b8803494a ]

Commit 5459c0b704 ("PCI/DPC: Quirk PIO log size for certain Intel Root
Ports") added quirks for Tiger and Alder Lake Root Ports but missed that
the same issue exists also in the previous generation, Ice Lake.

Apply the quirk for Ice Lake Root Ports as well.  This prevents kernel
complaints like:

  DPC: RP PIO log size 0 is invalid

and also enables the DPC driver to dump the RP PIO Log registers when DPC
is triggered.

[bhelgaas: add dmesg warning and RP PIO Log dump info]
Closes: https://bugzilla.kernel.org/show_bug.cgi?id=209943
Link: https://lore.kernel.org/r/20230511121905.73949-1-mika.westerberg@linux.intel.com
Reported-by: Mark Blakeney <mark.blakeney@bullet-systems.net>
Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Signed-off-by: Sasha Levin <sashal@kernel.org>
2023-06-21 16:00:52 +02:00
Alex Williamson
08e9653bb9 PCI/PM: Extend D3hot delay for NVIDIA HDA controllers
[ Upstream commit a5a6dd2624 ]

Assignment of NVIDIA Ampere-based GPUs have seen a regression since the
below referenced commit, where the reduced D3hot transition delay appears
to introduce a small window where a D3hot->D0 transition followed by a bus
reset can wedge the device.  The entire device is subsequently unavailable,
returning -1 on config space read and is unrecoverable without a host
reset.

This has been observed with RTX A2000 and A5000 GPU and audio functions
assigned to a Windows VM, where shutdown of the VM places the devices in
D3hot prior to vfio-pci performing a bus reset when userspace releases the
devices.  The issue has roughly a 2-3% chance of occurring per shutdown.

Restoring the HDA controller d3hot_delay to the effective value before the
below commit has been shown to resolve the issue.  NVIDIA confirms this
change should be safe for all of their HDA controllers.

Fixes: 3e347969a5 ("PCI/PM: Reduce D3hot delay with usleep_range()")
Link: https://lore.kernel.org/r/20230413194042.605768-1-alex.williamson@redhat.com
Reported-by: Zhiyi Guo <zhguo@redhat.com>
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Tarun Gupta <targupta@nvidia.com>
Cc: Abhishek Sahu <abhsahu@nvidia.com>
Cc: Tarun Gupta <targupta@nvidia.com>
Signed-off-by: Sasha Levin <sashal@kernel.org>
2023-05-11 23:03:29 +09:00
Mengyuan Lou
455ed25b59 PCI: Add ACS quirk for Wangxun NICs
[ Upstream commit a2b9b123cc ]

Wangxun has verified there is no peer-to-peer between functions for the
below selection of SFxxx, RP1000 and RP2000 NICS.  They may be
multi-function devices, but the hardware does not advertise ACS capability.

Add an ACS quirk for these devices so the functions can be in independent
IOMMU groups.

Link: https://lore.kernel.org/r/20230207102419.44326-1-mengyuanlou@net-swift.com
Signed-off-by: Mengyuan Lou <mengyuanlou@net-swift.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Signed-off-by: Sasha Levin <sashal@kernel.org>
2023-03-11 13:55:39 +01:00
Damien Le Moal
358dbd5751 PCI: Avoid FLR for AMD FCH AHCI adapters
commit 63ba51db24 upstream.

PCI passthrough to VMs does not work with AMD FCH AHCI adapters: the guest
OS fails to correctly probe devices attached to the controller due to FIS
communication failures:

  ata4: softreset failed (1st FIS failed)
  ...
  ata4.00: qc timeout after 5000 msecs (cmd 0xec)
  ata4.00: failed to IDENTIFY (I/O error, err_mask=0x4)

Forcing the "bus" reset method before unbinding & binding the adapter to
the vfio-pci driver solves this issue, e.g.:

  echo "bus" > /sys/bus/pci/devices/<ID>/reset_method

gives a working guest OS, indicating that the default FLR reset method
doesn't work correctly.

Apply quirk_no_flr() to AMD FCH AHCI devices to work around this issue.

Link: https://lore.kernel.org/r/20230128013951.523247-1-damien.lemoal@opensource.wdc.com
Reported-by: Niklas Cassel <niklas.cassel@wdc.com>
Signed-off-by: Damien Le Moal <damien.lemoal@opensource.wdc.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Cc: stable@vger.kernel.org
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2023-03-10 09:34:31 +01:00
Mika Westerberg
5459c0b704 PCI/DPC: Quirk PIO log size for certain Intel Root Ports
Some Root Ports on Intel Tiger Lake and Alder Lake systems support the RP
Extensions for DPC and the RP PIO Log registers but incorrectly advertise
an RP PIO Log Size of zero.  This means the kernel complains that:

  DPC: RP PIO log size 0 is invalid

and if DPC is triggered, the DPC driver will not dump the RP PIO Log
registers when it should.

This is caused by a BIOS bug and should be fixed the BIOS for future CPUs.

Add a quirk to set the correct RP PIO Log size for the affected Root Ports.

Link: https://bugzilla.kernel.org/show_bug.cgi?id=209943
Link: https://lore.kernel.org/r/20220816102042.69125-1-mika.westerberg@linux.intel.com
Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Kuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>
2022-09-27 18:13:18 -05:00
Bjorn Helgaas
c4f36c3ab0 Merge branch 'pci/header-cleanup-immutable'
- Remove pci_get_legacy_ide_irq(); use ATA_PRIMARY_IRQ() and
  ATA_SECONDARY_IRQ() instead (Stafford Horne)

- Remove isa_dma_bridge_buggy, except for x86_32, the only place it's used
  (Stafford Horne)

- Define ARCH_GENERIC_PCI_MMAP_RESOURCE for csky (Stafford Horne)

- Move common PCI definitions that arches sometimes override to
  asm-generic/pci.h (Stafford Horne)

- Include <linux/isa-dma.h> for 'isa_dma_bridge_buggy' when needed
  (bisection hole here) (Randy Dunlap)

* pci/header-cleanup-immutable:
  PCI: Stub __pci_ioport_map() for arches that don't support it at all
  x86/cyrix: include header linux/isa-dma.h
  asm-generic: Add new pci.h and use it
  csky: PCI: Define ARCH_GENERIC_PCI_MMAP_RESOURCE
  PCI: Move isa_dma_bridge_buggy out of asm/dma.h
  PCI: Remove pci_get_legacy_ide_irq() and asm-generic/pci.h
2022-08-04 11:46:53 -05:00
Bjorn Helgaas
64451ac83f Merge branch 'pci/ctrl/tegra194'
- Fix tegra_pcie_config_ep() power management in error path (Miaoqian Lin)

- Convert DT binding to json-schema (Vidya Sagar)

- Add DT bindings and driver support for Tegra234 Root Port and Endpoint
  mode (Vidya Sagar)

- Disable MSI for Tegra234 Root Ports so they use INTx for all events (PCIe
  doesn't allow mixing INTx and MSI/MSI-X) (Vidya Sagar)

- Search for Vendor-Specific RAS-DEC capability instead of hard-coding
  offset (Vidya Sagar)

- Fix unintentional APPL_INTR_STATUS_L0 value overwrite in Root Port
  interrupt handling (Vidya Sagar)

- Clear Bandwidth Management interrupt status bit to avoid interrupt storm
  (Vidya Sagar)

- Set default Max Payload Size to 256 bytes (Vidya Sagar)

- Fix offset when clearing bit in Data Link Feature capability (Vidya
  Sagar)

- Extend Endpoint mode support to devices beyond Controller-5 (Vidya Sagar)

* pci/ctrl/tegra194:
  PCI: tegra194: Add Tegra234 PCIe support
  PCI: tegra194: Extend Endpoint mode support
  PCI: tegra194: Fix link up retry sequence
  PCI: tegra194: Clean up the exit path for Endpoint mode
  PCI: tegra194: Enable support for 256 Byte payload
  PCI: tegra194: Clear bandwidth management status
  PCI: tegra194: Fix Root Port interrupt handling
  PCI: tegra194: Find RAS DES PCIe capability offset
  Revert "PCI: tegra194: Rename tegra_pcie_dw to tegra194_pcie"
  PCI: Disable MSI for Tegra234 Root Ports
  dt-bindings: PCI: tegra234: Add schema for tegra234 Endpoint mode
  dt-bindings: PCI: tegra234: Add schema for tegra234 Root Port mode
  dt-bindings: PCI: tegra194: Convert to json-schema
  PCI: tegra194: Fix PM error handling in tegra_pcie_config_ep()

# Conflicts:
#	drivers/pci/controller/dwc/pcie-designware.h
#	drivers/pci/controller/dwc/pcie-tegra194.c
2022-08-04 11:46:51 -05:00
Stafford Horne
abb4970ac3 PCI: Move isa_dma_bridge_buggy out of asm/dma.h
The isa_dma_bridge_buggy symbol is only used for x86_32, and only x86_32
platforms or quirks ever set it.

Add a new linux/isa-dma.h header that #defines isa_dma_bridge_buggy to 0
except on x86_32, where we keep it as a variable, and remove all the arch-
specific definitions.

[bhelgaas: commit log]
Suggested-by: Arnd Bergmann <arnd@arndb.de>
Suggested-by: Christoph Hellwig <hch@infradead.org>
Link: https://lore.kernel.org/r/20220722214944.831438-3-shorne@gmail.com
Signed-off-by: Stafford Horne <shorne@gmail.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Acked-by: Geert Uytterhoeven <geert@linux-m68k.org>
2022-07-22 17:24:47 -05:00
Vidya Sagar
bf32b8f952 PCI: Disable MSI for Tegra234 Root Ports
Tegra234 PCIe Root Ports don't generate MSI interrupts for PME and AER
events. Since PCIe spec (r6.0 sec 6.1.4.3) doesn't support using a mix of
INTx and MSI/MSI-X, MSI needs to be disabled to avoid Root Port service
drivers registering their respective ISRs with MSI interrupt and to let
only INTx be used for all events.

Link: https://lore.kernel.org/r/20220721142052.25971-8-vidyas@nvidia.com
Signed-off-by: Vidya Sagar <vidyas@nvidia.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
2022-07-22 17:14:56 -05:00
Pavan Chebbi
afd306a65c PCI: Add ACS quirk for Broadcom BCM5750x NICs
The Broadcom BCM5750x NICs may be multi-function devices.  They do not
advertise ACS capability. Peer-to-peer transactions are not possible
between the individual functions, so it is safe to treat them as fully
isolated.

Add an ACS quirk for these devices so the functions can be in independent
IOMMU groups and attached individually to userspace applications using
VFIO.

Link: https://lore.kernel.org/r/1654796507-28610-1-git-send-email-michael.chan@broadcom.com
Signed-off-by: Pavan Chebbi <pavan.chebbi@broadcom.com>
Signed-off-by: Michael Chan <michael.chan@broadcom.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
2022-06-09 13:57:15 -05:00
Mika Westerberg
03038d84ac PCI/ASPM: Make Intel DG2 L1 acceptable latency unlimited
Intel DG2 discrete graphics PCIe endpoints advertise L1 acceptable exit
latency to be < 1us even though they can actually tolerate unlimited exit
latencies just fine. Quirk the L1 acceptable exit latency for these
endpoints to be unlimited so ASPM L1 can be enabled.

[bhelgaas: use FIELD_GET/FIELD_PREP, wordsmith comment & commit log]
Link: https://lore.kernel.org/r/20220405093810.76613-1-mika.westerberg@linux.intel.com
Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
2022-04-07 12:00:36 -05:00
Linus Torvalds
148a650476 pci-v5.18-changes
-----BEGIN PGP SIGNATURE-----
 
 iQJIBAABCgAyFiEEgMe7l+5h9hnxdsnuWYigwDrT+vwFAmI7iOwUHGJoZWxnYWFz
 QGdvb2dsZS5jb20ACgkQWYigwDrT+vxkuhAAtJkVwfeyUjZ8sms+qWdZaucJmFF1
 PDeKy8O8upLzRRykdWoAOjKKVcCB9ohxBjPMco2oYNTmSozxeau8jjMA9OTQvTOS
 ZhDDoi49/vHRHuq3WIeAMCuk7tH3H1L3f0UHJxJ3H/oObQ+eMsitPcGFK+QrISDX
 pYokOnXZvf7BT7NpVtogSe2mhniOD1zQSicAMiH6WKNHHZcxewrzV9LP3MFOoBAr
 VMhlhzJbOp9spvCt7M1DycJEQ2RNe+wGLBFDalhPuprwnkNchRV+0AwWfD90zc9u
 h/0J8jkXfqS6QfSd/lOlTvI6kGsV8UKZEt4h4X/hlHFebFM5ktD9X7GmcoYUDFd9
 aHV3I/Jf62uGJ31IrT0V/cSYNlMO+IVFwXLGir4B1cFPOkzyIG/i60iV/C6bnnCa
 TCMH6vxalFycYaHBFqw/K/Dlq+mrAX74nQDfbk8y6rprczM1BN220Z8BkpG13TBu
 MxgCEul2/BJmNcPS1IWb/mCfBy+rdrVn2DZuID3J9KTwKNOUTIuAF0FuxLP4Bk4o
 sti3vKIXOcHnAcJB9tEnpEfstPv2JT13eWDIMmp/qCwqcujOvsg/DSYrx+8ogmBF
 DJ/sbPy3BdIOAeTgepWHAxYcv9SlZTGJGl+oaR1zV0qLBogyQUWZ9Ijx5aAEAw3j
 AJicpdk3BkH3LC8=
 =5Q9H
 -----END PGP SIGNATURE-----

Merge tag 'pci-v5.18-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/helgaas/pci

Pull pci updates from Bjorn Helgaas:
 "Enumeration:
   - Move the VGA arbiter from drivers/gpu to drivers/pci because it's
     PCI-specific, not GPU-specific (Bjorn Helgaas)
   - Select the default VGA device consistently whether it's enumerated
     before or after VGA arbiter init, which fixes arches that enumerate
     PCI devices late (Huacai Chen)

  Resource management:
   - Support BAR sizes up to 8TB (Dongdong Liu)

  PCIe native device hotplug:
   - Fix "Command Completed" tracking to avoid spurious timouts when
     powering off empty slots (Liguang Zhang)
   - Quirk Qualcomm devices that don't implement Command Completed
     correctly, again to avoid spurious timeouts (Manivannan Sadhasivam)

  Peer-to-peer DMA:
   - Add Intel 3rd Gen Intel Xeon Scalable Processors to whitelist
     (Michael J. Ruhl)

  APM X-Gene PCIe controller driver:
   - Revert generic DT parsing changes that broke some machines in the
     field (Marc Zyngier)

  Freescale i.MX6 PCIe controller driver:
   - Allow controller probe to succeed even when no devices currently
     present to allow hot-add later (Fabio Estevam)
   - Enable power management on i.MX6QP (Richard Zhu)
   - Assert CLKREQ# on i.MX8MM so enumeration doesn't hang when no
     device is connected (Richard Zhu)

  Marvell Aardvark PCIe controller driver:
   - Fix MSI and MSI-X support (Marek Behún, Pali Rohár)
   - Add support for ERR and PME interrupts (Pali Rohár)

  Marvell MVEBU PCIe controller driver:
   - Add DT binding and support for "num-lanes" (Pali Rohár)
   - Add support for INTx interrupts (Pali Rohár)

  Microsoft Hyper-V host bridge driver:
   - Avoid unnecessary hypercalls when unmasking IRQs on ARM64 (Boqun
     Feng)

  Qualcomm PCIe controller driver:
   - Add SM8450 DT binding and driver support (Dmitry Baryshkov)

  Renesas R-Car PCIe controller driver:
   - Help the controller get to the L1 state since the hardware can't do
     it on its own (Marek Vasut)
   - Return PCI_ERROR_RESPONSE (~0) for reads that fail on PCIe (Marek
     Vasut)

  SiFive FU740 PCIe controller driver:
   - Drop redundant '-gpios' from DT GPIO lookup (Ben Dooks)
   - Force 2.5GT/s for initial device probe (Ben Dooks)

  Socionext UniPhier Pro5 controller driver:
   - Add NX1 DT binding and driver support (Kunihiko Hayashi)

  Synopsys DesignWare PCIe controller driver:
   - Restore MSI configuration so MSI works after resume (Jisheng
     Zhang)"

* tag 'pci-v5.18-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/helgaas/pci: (94 commits)
  x86/PCI: Add #includes to asm/pci_x86.h
  PCI: ibmphp: Remove unused assignments
  PCI: cpqphp: Remove unused assignments
  PCI: fu740: Remove unused assignments
  PCI: kirin: Remove unused assignments
  PCI: Remove unused assignments
  PCI: Declare pci_filp_private only when HAVE_PCI_MMAP
  PCI: Avoid broken MSI on SB600 USB devices
  PCI: fu740: Force 2.5GT/s for initial device probe
  PCI: xgene: Revert "PCI: xgene: Fix IB window setup"
  PCI: xgene: Revert "PCI: xgene: Use inbound resources for setup"
  PCI: imx6: Assert i.MX8MM CLKREQ# even if no device present
  PCI: imx6: Invoke the PHY exit function after PHY power off
  PCI: rcar: Use PCI_SET_ERROR_RESPONSE after read which triggered an exception
  PCI: rcar: Finish transition to L1 state in rcar_pcie_config_access()
  PCI: dwc: Restore MSI Receiver mask during resume
  PCI: fu740: Drop redundant '-gpios' from DT GPIO lookup
  PCI/VGA: Replace full MIT license text with SPDX identifier
  PCI/VGA: Use unsigned format string to print lock counts
  PCI/VGA: Log bridge control messages when adding devices
  ...
2022-03-25 13:02:05 -07:00
Bjorn Helgaas
63cd736f44 PCI: Avoid broken MSI on SB600 USB devices
Some ATI SB600 USB adapters advertise MSI, but if INTx is disabled by
setting PCI_COMMAND_INTX_DISABLE, MSI doesn't work either.  The PCI/PCIe
specs do not require software to set PCI_COMMAND_INTX_DISABLE when enabling
MSI, but Linux has done that for many years.

Mick reported that 306c54d0ed ("usb: hcd: Try MSI interrupts on PCI
devices") broke these devices.  Prior to 306c54d0ed, they used INTx.
Starting with 306c54d0ed, they use MSI, and and the fact that Linux sets
PCI_COMMAND_INTX_DISABLE means both INTx and MSI are disabled on these
devices.

Avoid this SB600 defect by disabling MSI so we use INTx as before.

Fixes: 306c54d0ed ("usb: hcd: Try MSI interrupts on PCI devices")
Link: https://lore.kernel.org/r/20220321183446.1108325-1-helgaas@kernel.org
Link: https://bugzilla.kernel.org/show_bug.cgi?id=215690
Link: https://lore.kernel.org/all/PxIByDyBRcsbpcmVhGSNDFAoUcMmb78ctXCkw6fbpx25TGlCHvA6SJjjFkNr1FfQZMntYPTNyvEnblxzAZ8a6jP9ddLpKeCN6Chi_2FuexU=@protonmail.com/
Link: https://lore.kernel.org/r/20220314101448.90074-1-andriy.shevchenko@linux.intel.com
BugLink: https://lore.kernel.org/all/20200702143045.23429-1-andriy.shevchenko@linux.intel.com/
Reported-by: Mick Lorain <micklorain@protonmail.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
2022-03-21 15:23:14 -05:00
Alex Deucher
3f1271b54e PCI: Mark all AMD Navi10 and Navi14 GPU ATS as broken
There are enough VBIOS escapes without the proper workaround that some
users still hit this.  Microsoft never productized ATS on Windows so OEM
platforms that were Windows-only didn't always validate ATS.

The advantages of ATS are not worth it compared to the potential
instabilities on harvested boards.  Disable ATS on all Navi10 and Navi14
boards.

Symptoms include:

  amdgpu 0000:07:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0007 address=0xffffc02000 flags=0x0000]
  AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.0 domain=0x0007 address=0xffffc02000 flags=0x0000]
  [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring sdma0 timeout, signaled seq=6047, emitted seq=6049
  amdgpu 0000:07:00.0: amdgpu: GPU reset begin!
  amdgpu 0000:07:00.0: amdgpu: GPU reset succeeded, trying to resume
  amdgpu 0000:07:00.0: [drm:amdgpu_ring_test_helper [amdgpu]] *ERROR* ring sdma0 test failed (-110)
  [drm:amdgpu_device_ip_resume_phase2 [amdgpu]] *ERROR* resume of IP block <sdma_v4_0> failed -110
  amdgpu 0000:07:00.0: amdgpu: GPU reset(1) failed

Related commits:

  e8946a53e2 ("PCI: Mark AMD Navi14 GPU ATS as broken")
  a2da5d8cc0 ("PCI: Mark AMD Raven iGPU ATS as broken in some platforms")
  45beb31d3a ("PCI: Mark AMD Navi10 GPU rev 0x00 ATS as broken")
  5e89cd303e ("PCI: Mark AMD Navi14 GPU rev 0xc5 ATS as broken")
  d28ca864c4 ("PCI: Mark AMD Stoney Radeon R7 GPU ATS as broken")
  9b44b0b09d ("PCI: Mark AMD Stoney GPU ATS as broken")

[bhelgaas: add symptoms and related commits]
Bug: https://gitlab.freedesktop.org/drm/amd/-/issues/1760
Link: https://lore.kernel.org/r/20220222160801.841643-1-alexander.deucher@amd.com
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Acked-by: Christian König <christian.koenig@amd.com>
Acked-by: Guchun Chen <guchun.chen@amd.com>
2022-02-23 12:33:32 -06:00
Linus Torvalds
d0a231f01e pci-v5.17-changes
-----BEGIN PGP SIGNATURE-----
 
 iQJIBAABCgAyFiEEgMe7l+5h9hnxdsnuWYigwDrT+vwFAmHgpugUHGJoZWxnYWFz
 QGdvb2dsZS5jb20ACgkQWYigwDrT+vz59g//eWRLb0j2Vgv84ZH4x1iv6MaBboQr
 2wScnfoN+MIoh+tuM4kRak15X4nB8rJhNZZCzesMUN6PeZvrkoPo4sz/xdzIrA/N
 qY3h8NZ3nC4yCvs/tGem0zZUcSCJsxUAD0eegyMSa142xGIOQTHBSJRflR9osKSo
 bnQlKTkugV8t4kD7NlQ5M3HzN3R+mjsII5JNzCqv2XlzAZG3D8DhPyIpZnRNAOmW
 KiHOVXvQOocfUlvSs5kBlhgR1HgJkGnruCrJ1iDCWQH1Zk0iuVgoZWgVda6Cs3Xv
 gcTJLB7VoSdNZKnct9aMNYPKziHkYc7clilPeDsJs5TlSv3kKERzLj6c/5ZAxFWN
 +RsH+zYHDXJSsL/w0twPnaF5WCuVYUyrs3UiSjUvShKl1T9k9J+Jo8zwUUZx8Xb0
 qXX8jRGMHolBGwPXm2fHEb4bwTUI8emPj29qK4L96KsQ3zKXWB8eGSosxUP52Tti
 RR2WZjkvwlREZCJp6jSEJYkhzoEaVAm8CjKpKUuneX9WcUOsMBSs9k7EXbUy7JeM
 hq5Keuqa8PZo/IK2DYYAchNnBJUDMsWJeduBW12qSmx3J+9victP2qOFu+9skP0a
 85xlO6Cx8beiQh+XnY7jyROvIFuxTnGKHgkq/89Ham/whEzdJ+GRIiYB218kLLCW
 ILdas3C2iiGz99I=
 =Vgg4
 -----END PGP SIGNATURE-----

Merge tag 'pci-v5.17-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/helgaas/pci

Pull pci updates from Bjorn Helgaas:
 "Enumeration:
   - Use pci_find_vsec_capability() instead of open-coding it (Andy
     Shevchenko)
   - Convert pci_dev_present() stub from macro to static inline to avoid
     'unused variable' errors (Hans de Goede)
   - Convert sysfs slot attributes from default_attrs to default_groups
     (Greg Kroah-Hartman)
   - Use DWORD accesses for LTR, L1 SS to avoid BayHub OZ711LV2 erratum
     (Rajat Jain)
   - Remove unnecessary initialization of static variables (Longji Guo)

  Resource management:
   - Always write Intel I210 ROM BAR on update to work around device
     defect (Bjorn Helgaas)

  PCIe native device hotplug:
   - Fix pciehp lockdep errors on Thunderbolt undock (Hans de Goede)
   - Fix infinite loop in pciehp IRQ handler on power fault (Lukas
     Wunner)

  Power management:
   - Convert amd64-agp, sis-agp, via-agp from legacy PCI power
     management to generic power management (Vaibhav Gupta)

  IOMMU:
   - Add function 1 DMA alias quirk for Marvell 88SE9125 SATA controller
     so it can work with an IOMMU (Yifeng Li)

  Error handling:
   - Add PCI_ERROR_RESPONSE and related definitions for signaling and
     checking for transaction errors on PCI (Naveen Naidu)
   - Fabricate PCI_ERROR_RESPONSE data (~0) in config read wrappers,
     instead of in host controller drivers, when transactions fail on
     PCI (Naveen Naidu)
   - Use PCI_POSSIBLE_ERROR() to check for possible failure of config
     reads (Naveen Naidu)

  Peer-to-peer DMA:
   - Add Logan Gunthorpe as P2PDMA maintainer (Bjorn Helgaas)

  ASPM:
   - Calculate link L0s and L1 exit latencies when needed instead of
     caching them (Saheed O. Bolarinwa)
   - Calculate device L0s and L1 acceptable exit latencies when needed
     instead of caching them (Saheed O. Bolarinwa)
   - Remove struct aspm_latency since it's no longer needed (Saheed O.
     Bolarinwa)

  APM X-Gene PCIe controller driver:
   - Fix IB window setup, which was broken by the fact that IB resources
     are now sorted in address order instead of DT dma-ranges order (Rob
     Herring)

  Apple PCIe controller driver:
   - Enable clock gating to save power (Hector Martin)
   - Fix REFCLK1 enable/poll logic (Hector Martin)

  Broadcom STB PCIe controller driver:
   - Declare bitmap correctly for use by bitmap interfaces (Christophe
     JAILLET)
   - Clean up computation of legacy and non-legacy MSI bitmasks (Florian
     Fainelli)
   - Update suspend/resume/remove error handling to warn about errors
     and not fail the operation (Jim Quinlan)
   - Correct the "pcie" and "msi" interrupt descriptions in DT binding
     (Jim Quinlan)
   - Add DT bindings for endpoint voltage regulators (Jim Quinlan)
   - Split brcm_pcie_setup() into two functions (Jim Quinlan)
   - Add mechanism for turning on voltage regulators for connected
     devices (Jim Quinlan)
   - Turn voltage regulators for connected devices on/off when bus is
     added or removed (Jim Quinlan)
   - When suspending, don't turn off voltage regulators for wakeup
     devices (Jim Quinlan)

  Freescale i.MX6 PCIe controller driver:
   - Add i.MX8MM support (Richard Zhu)

  Freescale Layerscape PCIe controller driver:
   - Use DWC common ops instead of layerscape-specific link-up functions
     (Hou Zhiqiang)

  Intel VMD host bridge driver:
   - Honor platform ACPI _OSC feature negotiation for Root Ports below
     VMD (Kai-Heng Feng)
   - Add support for Raptor Lake SKUs (Karthik L Gopalakrishnan)
   - Reset everything below VMD before enumerating to work around
     failure to enumerate NVMe devices when guest OS reboots (Nirmal
     Patel)

  Bridge emulation (used by Marvell Aardvark and MVEBU):
   - Make emulated ROM BAR read-only by default (Pali Rohár)
   - Make some emulated legacy PCI bits read-only for PCIe devices (Pali
     Rohár)
   - Update reserved bits in emulated PCIe Capability (Pali Rohár)
   - Allow drivers to emulate different PCIe Capability versions (Pali
     Rohár)
   - Set emulated Capabilities List bit for all PCIe devices, since they
     must have at least a PCIe Capability (Pali Rohár)

  Marvell Aardvark PCIe controller driver:
   - Add bridge emulation definitions for PCIe DEVCAP2, DEVCTL2,
     DEVSTA2, LNKCAP2, LNKCTL2, LNKSTA2, SLTCAP2, SLTCTL2, SLTSTA2 (Pali
     Rohár)
   - Add aardvark support for DEVCAP2, DEVCTL2, LNKCAP2 and LNKCTL2
     registers (Pali Rohár)
   - Clear all MSIs at setup to avoid spurious interrupts (Pali Rohár)
   - Disable bus mastering when unbinding host controller driver (Pali
     Rohár)
   - Mask all interrupts when unbinding host controller driver (Pali
     Rohár)
   - Fix memory leak in host controller unbind (Pali Rohár)
   - Assert PERST# when unbinding host controller driver (Pali Rohár)
   - Disable link training when unbinding host controller driver (Pali
     Rohár)
   - Disable common PHY when unbinding host controller driver (Pali
     Rohár)
   - Fix resource type checking to check only IORESOURCE_MEM, not
     IORESOURCE_MEM_64, which is a flavor of IORESOURCE_MEM (Pali Rohár)

  Marvell MVEBU PCIe controller driver:
   - Implement pci_remap_iospace() for ARM so mvebu can use
     devm_pci_remap_iospace() instead of the previous ARM-specific
     pci_ioremap_io() interface (Pali Rohár)
   - Use the standard pci_host_probe() instead of the device-specific
     mvebu_pci_host_probe() (Pali Rohár)
   - Replace all uses of ARM-specific pci_ioremap_io() with the ARM
     implementation of the standard pci_remap_iospace() interface and
     remove pci_ioremap_io() (Pali Rohár)
   - Skip initializing invalid Root Ports (Pali Rohár)
   - Check for errors from pci_bridge_emul_init() (Pali Rohár)
   - Ignore any bridges at non-zero function numbers (Pali Rohár)
   - Return ~0 data for invalid config read size (Pali Rohár)
   - Disallow mapping interrupts on emulated bridges (Pali Rohár)
   - Clear Root Port Memory & I/O Space Enable and Bus Master Enable at
     initialization (Pali Rohár)
   - Make type bits in Root Port I/O Base register read-only (Pali
     Rohár)
   - Disable Root Port windows when base/limit set to invalid values
     (Pali Rohár)
   - Set controller to Root Complex mode (Pali Rohár)
   - Set Root Port Class Code to PCI Bridge (Pali Rohár)
   - Update emulated Root Port secondary bus numbers to better reflect
     the actual topology (Pali Rohár)
   - Add PCI_BRIDGE_CTL_BUS_RESET support to emulated Root Ports so
     pci_reset_secondary_bus() can reset connected devices (Pali Rohár)
   - Add PCI_EXP_DEVCTL Error Reporting Enable support to emulated Root
     Ports (Pali Rohár)
   - Add PCI_EXP_RTSTA PME Status bit support to emulated Root Ports
     (Pali Rohár)
   - Add DEVCAP2, DEVCTL2 and LNKCTL2 support to emulated Root Ports on
     Armada XP and newer devices (Pali Rohár)
   - Export mvebu-mbus.c symbols to allow pci-mvebu.c to be a module
     (Pali Rohár)
   - Add support for compiling as a module (Pali Rohár)

  MediaTek PCIe controller driver:
   - Assert PERST# for 100ms to allow power and clock to stabilize
     (qizhong cheng)

  MediaTek PCIe Gen3 controller driver:
   - Disable Mediatek DVFSRC voltage request since lack of DVFSRC to
     respond to the request causes failure to exit L1 PM Substate
     (Jianjun Wang)

  MediaTek MT7621 PCIe controller driver:
   - Declare mt7621_pci_ops static (Sergio Paracuellos)
   - Give pcibios_root_bridge_prepare() access to host bridge windows
     (Sergio Paracuellos)
   - Move MIPS I/O coherency unit setup from driver to
     pcibios_root_bridge_prepare() (Sergio Paracuellos)
   - Add missing MODULE_LICENSE() (Sergio Paracuellos)
   - Allow COMPILE_TEST for all arches (Sergio Paracuellos)

  Microsoft Hyper-V host bridge driver:
   - Add hv-internal interfaces to encapsulate arch IRQ dependencies
     (Sunil Muthuswamy)
   - Add arm64 Hyper-V vPCI support (Sunil Muthuswamy)

  Qualcomm PCIe controller driver:
   - Undo PM setup in qcom_pcie_probe() error handling path (Christophe
     JAILLET)
   - Use __be16 type to store return value from cpu_to_be16()
     (Manivannan Sadhasivam)
   - Constify static dw_pcie_ep_ops (Rikard Falkeborn)

  Renesas R-Car PCIe controller driver:
   - Fix aarch32 abort handler so it doesn't check the wrong bus clock
     before accessing the host controller (Marek Vasut)

  TI Keystone PCIe controller driver:
   - Add register offset for ti,syscon-pcie-id and ti,syscon-pcie-mode
     DT properties (Kishon Vijay Abraham I)

  MicroSemi Switchtec management driver:
   - Add Gen4 automotive device IDs (Kelvin Cao)
   - Declare state_names[] as static so it's not allocated and
     initialized for every call (Kelvin Cao)

  Host controller driver cleanups:
   - Use of_device_get_match_data(), not of_match_device(), when we only
     need the device data in altera, artpec6, cadence, designware-plat,
     dra7xx, keystone, kirin (Fan Fei)
   - Drop pointless of_device_get_match_data() cast in j721e (Bjorn
     Helgaas)
   - Drop redundant struct device * from j721e since struct cdns_pcie
     already has one (Bjorn Helgaas)
   - Rename driver structs to *_pcie in intel-gw, iproc, ls-gen4,
     mediatek-gen3, microchip, mt7621, rcar-gen2, tegra194, uniphier,
     xgene, xilinx, xilinx-cpm for consistency across drivers (Fan Fei)
   - Fix invalid address space conversions in hisi, spear13xx (Bjorn
     Helgaas)

  Miscellaneous:
   - Sort Intel Device IDs by value (Andy Shevchenko)
   - Change Capability offsets to hex to match spec (Baruch Siach)
   - Correct misspellings (Krzysztof Wilczyński)
   - Terminate statement with semicolon in pci_endpoint_test.c (Ming
     Wang)"

* tag 'pci-v5.17-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/helgaas/pci: (151 commits)
  PCI: mt7621: Allow COMPILE_TEST for all arches
  PCI: mt7621: Add missing MODULE_LICENSE()
  PCI: mt7621: Move MIPS setup to pcibios_root_bridge_prepare()
  PCI: Let pcibios_root_bridge_prepare() access bridge->windows
  PCI: mt7621: Declare mt7621_pci_ops static
  PCI: brcmstb: Do not turn off WOL regulators on suspend
  PCI: brcmstb: Add control of subdevice voltage regulators
  PCI: brcmstb: Add mechanism to turn on subdev regulators
  PCI: brcmstb: Split brcm_pcie_setup() into two funcs
  dt-bindings: PCI: Add bindings for Brcmstb EP voltage regulators
  dt-bindings: PCI: Correct brcmstb interrupts, interrupt-map.
  PCI: brcmstb: Fix function return value handling
  PCI: brcmstb: Do not use __GENMASK
  PCI: brcmstb: Declare 'used' as bitmap, not unsigned long
  PCI: hv: Add arm64 Hyper-V vPCI support
  PCI: hv: Make the code arch neutral by adding arch specific interfaces
  PCI: pciehp: Use down_read/write_nested(reset_lock) to fix lockdep errors
  x86/PCI: Remove initialization of static variables to false
  PCI: Use DWORD accesses for LTR, L1 SS to avoid erratum
  misc: pci_endpoint_test: Terminate statement with semicolon
  ...
2022-01-16 08:08:11 +02:00
Bjorn Helgaas
da43f08db2 Merge branch 'pci/misc'
- Sort Intel Device IDs by value (Andy Shevchenko)

- Change Capability offsets to hex to match spec (Baruch Siach)

- Correct misspellings (Krzysztof Wilczyński)

- Terminate statement with semicolon in pci_endpoint_test.c (Ming Wang)

* pci/misc:
  misc: pci_endpoint_test: Terminate statement with semicolon
  PCI: Correct misspelled words
  PCI: Change capability register offsets to hex
  PCI: Sort Intel Device IDs by value
2022-01-13 09:57:52 -06:00
Bjorn Helgaas
d03f92c43f Merge branch 'pci/virtualization'
- Add function 1 DMA alias quirk for Marvell 88SE9125 SATA controller so it
  can work with an IOMMU (Yifeng Li)

* pci/virtualization:
  PCI: Add function 1 DMA alias quirk for Marvell 88SE9125 SATA controller
2022-01-13 09:57:45 -06:00
Bjorn Helgaas
54f98a8b13 Merge branch 'pci/switchtec'
- Add Gen4 automotive device IDs (Kelvin Cao)

- Declare state_names[] as static so it's not allocated and initialized for
  every call (Kelvin Cao)

* pci/switchtec:
  PCI/switchtec: Declare local state_names[] as static
  PCI/switchtec: Add Gen4 automotive device IDs
2022-01-13 09:57:45 -06:00
Bjorn Helgaas
500b55b05d PCI: Work around Intel I210 ROM BAR overlap defect
Per PCIe r5, sec 7.5.1.2.4, a device must not claim accesses to its
Expansion ROM unless both the Memory Space Enable and the Expansion ROM
Enable bit are set.  But apparently some Intel I210 NICs don't work
correctly if the ROM BAR overlaps another BAR, even if the Expansion ROM is
disabled.

Michael reported that on a Kontron SMARC-sAL28 ARM64 system with U-Boot
v2021.01-rc3, the ROM BAR overlaps BAR 3, and networking doesn't work at
all:

  BAR 0: 0x40000000 (32-bit, non-prefetchable) [size=1M]
  BAR 3: 0x40200000 (32-bit, non-prefetchable) [size=16K]
  ROM:   0x40200000 (disabled) [size=1M]

  NETDEV WATCHDOG: enP2p1s0 (igb): transmit queue 0 timed out
  Hardware name: Kontron SMARC-sAL28 (Single PHY) on SMARC Eval 2.0 carrier (DT)
  igb 0002:01:00.0 enP2p1s0: Reset adapter

Previously, pci_std_update_resource() wrote the assigned ROM address to the
BAR only when the ROM was enabled.  This meant that the I210 ROM BAR could
be left with an address assigned by firmware, which might overlap with
other BARs.

Quirk these I210 devices so pci_std_update_resource() always writes the
assigned address to the ROM BAR, whether or not the ROM is enabled.

Link: https://lore.kernel.org/r/20211223163754.GA1267351@bhelgaas
Link: https://lore.kernel.org/r/20201230185317.30915-1-michael@walle.cc
Link: https://bugzilla.kernel.org/show_bug.cgi?id=211105
Reported-by: Michael Walle <michael@walle.cc>
Tested-by: Michael Walle <michael@walle.cc>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
2022-01-11 09:33:10 -06:00
Krzysztof Wilczyński
ccd36795be PCI: Correct misspelled words
Fix a number of misspelled words, and while at it, correct two phrases used
to indicate a status of an operation where words used have been cleverly
truncated and thus always trigger a spellchecking error while performing a
static code analysis over the PCI tree.

[bhelgaas: reverse sense of quirk ternary]
Link: https://lore.kernel.org/r/20220107225942.121484-1-kw@linux.com
Signed-off-by: Krzysztof Wilczyński <kw@linux.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
2022-01-07 20:43:23 -06:00
Yifeng Li
e445375882 PCI: Add function 1 DMA alias quirk for Marvell 88SE9125 SATA controller
Like other SATA controller chips in the Marvell 88SE91xx series, the
Marvell 88SE9125 has the same DMA requester ID hardware bug that prevents
it from working under IOMMU.  Add it to the list of devices that need the
quirk.

Without this patch, device initialization fails with DMA errors:

  ata8: softreset failed (1st FIS failed)
  DMAR: DRHD: handling fault status reg 2
  DMAR: [DMA Write NO_PASID] Request device [03:00.1] fault addr 0xfffc0000 [fault reason 0x02] Present bit in context entry is clear
  DMAR: DRHD: handling fault status reg 2
  DMAR: [DMA Read NO_PASID] Request device [03:00.1] fault addr 0xfffc0000 [fault reason 0x02] Present bit in context entry is clear

After applying the patch, the controller can be successfully initialized:

  ata8: SATA link up 1.5 Gbps (SStatus 113 SControl 330)
  ata8.00: ATAPI: PIONEER BD-RW   BDR-207M, 1.21, max UDMA/100
  ata8.00: configured for UDMA/100
  scsi 7:0:0:0: CD-ROM            PIONEER  BD-RW   BDR-207M 1.21 PQ: 0 ANSI: 5

Link: https://lore.kernel.org/r/YahpKVR+McJVDdkD@work
Reported-by: Sam Bingner <sam@bingner.com>
Tested-by: Sam Bingner <sam@bingner.com>
Tested-by: Yifeng Li <tomli@tomli.me>
Signed-off-by: Yifeng Li <tomli@tomli.me>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Krzysztof Wilczyński <kw@linux.com>
Cc: stable@vger.kernel.org
2021-12-15 14:28:18 -06:00
Heikki Krogerus
0c9e032a45 PCI: Convert to device_create_managed_software_node()
In quirk_huawei_pcie_sva(), device_add_properties() is used to
inject additional device properties, but there is no
device_remove_properties() call anywhere to remove those
properties. The assumption is most likely that the device is
never removed, and the properties therefore do not also never
need to be removed.

Even though it is unlikely that the device is ever removed in
this case, it is safer to make sure that the properties are
also removed if the device ever does get unregistered.

To achieve this, instead of adding a separate quirk for the
case of device removal where device_remove_properties() is
called, using device_create_managed_software_node() instead of
device_add_properties(). Both functions create a software node
(a type of fwnode) that holds the device properties, which is
then assigned to the device very much the same way.

The difference between the two functions is, that
device_create_managed_software_node() guarantees that the
software node (together with the properties) is removed when
the device is removed. The function device_add_property() does
not guarantee that, so the properties added with it should
always be removed with device_remove_properties().

Reviewed-by: Andy Shevchenko <andy.shevchenko@gmail.com>
Acked-by: Zhangfei Gao <zhangfei.gao@linaro.org>
Acked-by: Bjorn Helgaas <bhelgaas@google.com>
Signed-off-by: Heikki Krogerus <heikki.krogerus@linux.intel.com>
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
2021-11-24 15:28:01 +01:00
Kelvin Cao
bb17b15813 PCI/switchtec: Add Gen4 automotive device IDs
Advertise support of the Gen4 automotive variants in module's device ID
table and add the same IDs to the list of switchtec quirks.

Link: https://lore.kernel.org/r/20211119003803.2333-2-kelvin.cao@microchip.com
Signed-off-by: Kelvin Cao <kelvin.cao@microchip.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Logan Gunthorpe <logang@deltatee.com>
2021-11-19 12:13:50 -06:00